dlenski/wtf

Whitespace Total Fixer

Whitespace Total Fixer

Identifies and/or fixes inconsistent whitespace and line endings in text files, so that they don't clog up your commits to version control systems like Git, Mercurial, or Subversion.

• Whitespace Total Fixer
  • Why you should use it
  • Quick Install
  • How to use it
  • Git pre-commit hooks
    • Careful version
    • Simple version
• Exciting origin story
• Whitespace issues addressed
• Reporting
• Todo
• Bugs
  • Bash completion
• Author
• License

Why you should use it:

• It's like an incomprehensible shell-script one-liner (e.g. sed -e 's/LINENOISE/'), but way better
• It's similar to git stripspace, but more flexible and detailed.
• wtf.py is a simple Python script with no dependencies beyond the standard Python library. It should run correctly with either Python 2.7 or 3.x.

Quick Install

curl https://raw.githubusercontent.com/dlenski/wtf/master/wtf.py > ~/bin/wtf.py && chmod 0755 !#:3

How to use it

See below for options to control exactly which whitespace issues it fixes, but here are some examples:

# consistent whitespace from programs that generate text files
dump_database_schema | wtf.py -o clean_output.sql

# in-place editing
wtf.py -i file1.txt file2.txt file3.txt
wtf.py -I.bak file1.txt file2.txt file3.txt # ditto, with backups

# summarize a bunch of files without actually editing them (-0)
find . -name "*.txt" -exec wtf.py -0 {} \;

# more advanced: find interesting text files. pass options and a directory
wtf-find() {
    find "${@: -1}" -not \( -name .svn -prune -o -name .git -prune \
         -o -name .hg -prune \) -type f -exec bash -c \
         'grep -Il "" "$1" &>/dev/null && wtf.py '"${*:1:$#-1}"' "$1"' _ {} \;
}
wtf-find -0 .

# exit status
wtf.py file1.txt file2.txt file3.txt > /dev/null
if (( $? == 10 )); then
  echo "issues fixed"
elif (( $? == 20 )); then
  echo "unfixed issues!"
fi

Git pre-commit hooks

You can use Git pre-commit hooks to automatically run wtf and cleanup whitespace in your repository prior to every commit. (You can bypass the hook, however, with git commit [--no-verify|-n].)

Careful version

The pre-commit.careful hook will run wtf, with the default options, on all the to-be-committed text files. You can easily use it as-is by creating a symlink, for example by running the following in your repository's working directory:

ln -s /path/to/wtf/source/pre-commit.careful .git/hooks/pre-commit

If you want to modify it, create the file .git/hooks/pre-commit in your repository's working directory, and ensure that it is executable (with chmod +x .git/hooks/pre-commit).

Features:

1. This hook only modifies your commits, and does not touch Git's working tree. If you understand and use Git's index ("staging area"), this should make sense. It will play nicely with git add --patch.
2. It skips over files marked as binary or non-text (-text) according to your repository's https://git-scm.com/docs/gitattributes
3. Note that when a file changes in your commit but remains unchanged in your working tree, the file will be marked modified by git status immediately after the commit.

Simple version

This version modifies Git's working tree as well as your commits. This version will not play nicely with git add --patch.

This version will run wtf -i, with any other options you add to wtf_options, on all the to-be-committed text files. They will be cleaned up in the commit, as well as in your working tree.

#!/bin/sh
wtf_options=''

# get a list of to-be-committed filepaths, EXCLUDING files considered
# by Git to be binary
committees=$(git diff --cached --numstat --diff-filter=ACMRTU|egrep -v ^-|cut -f3-)

# Run Whitespace Total Fixer in-place, and re-add files modified by it
for committee in $committees
do
	wtf -i $wtf_options "$committee" || git add "$committee"
done

Exciting origin story

One day at work, I spent way too much time wrangling commits laden with whitespace issues generated by recalcitrant text-editing tools on multiple platforms.

That evening, I went home and spent way too much time writing this program instead!

Whitespace issues addressed

WTF currently fixes, or simply reports, a few common types of whitespace issues. Most of these issues offer three possible command-line options enabling the user to fix, report, or ignore the issue.

• Remove trailing spaces at the ends of lines (default is fix):

    -t, --trail-space
    -T, --report-trail-space
    -It, --ignore-trail-space
  

• Remove blank lines at the ends of files (default is fix):

    -b, --eof-blanks
    -B, --report-eof-blanks
    -Ib, --ignore-eof-blanks
  

• Ensure that a new-line character appears at the end of the file (default is fix):

    -n, --eof-newl
    -N, --report-eof-newl
    -In, --ignore-eof-newl
  

• Make sure that all lines have matching EOL markers (that is, no mixing of lf/crlf). Default is to fix non-matching EOL characters by making them all the same as the first line of the file. The desired EOL markers can also be set to a specific value (lf/crlf/native), in which case all lines will unconditionally receive this marker.

    -E ENDING, --coerce-eol ENDING
    -e ENDING, --expect-eol ENDING
    -Ie, --ignore-eol
  

• Check for spaces followed by tabs in the whitespace at the beginning of a line; fixing this condition requires setting either --change-tabs or --change-spaces. The default is to report and warn:

    -s, --tab-space-mix
    -S, --report-tab-space-mix
    -Is, --ignore-tab-space-mix
  

• Change tabs in the whitespace at the beginning of a line to the specified number of spaces (default is not to change tabs at all). This option will not touch leading tabs when they are mixed with spaces, unless --tab-space-mix is also specified.

    -x SPACES, --change-tabs SPACES
  

• Change the specified number of consecutive spaces in the whitespace at the beginning of a line to tabs (default is not to change spaces at all). This option will not touch leading tabs when they are mixed with spaces, unless --tab-space-mix is also specified.

    -y SPACES, --change-spaces SPACES
  

Reporting

Unless the -q/--quiet option is used, WTF will summarize each file processed in which any whitespace issues were found and/or fixed. With -v it will also report issue-free files.

$ wtf -0 nightmare.txt    # -0 is equivalent to > /dev/null
nightmare.txt LINE 8: WARNING: spaces followed by tabs in whitespace at beginning of line
nightmare.txt:
    CHOPPED 1 lines with trailing space
    CHOPPED 0 blank lines at EOF
    ADDED newline at EOF
    CHANGED 1 line endings which didn't match crlf from first line
    WARNED ABOUT 1 lines with tabs/spaces mix

WTF will return the following exit codes on successful operation:

• 0: no issues seen (or -X/--no-exit-codes specified)
• 10: issues fixed
• 20: unfixed issues seen

Todo

• Stability tests?
• Unicode tests?

Bugs

Corrupts source code files written in the Whitespace programming language.

Anything else?

Bash completion

If you are having problems with tab completion in Bash, it is probably because of default completion rules for the unrelated utility wtf(6). You can override this by adding the following to your ~/.bashrc:

complete -o default -o bashdefault wtf

Author

© Daniel Lenski <[email protected]> (2014-2020)

License

GPL v3 or later

wtf's Issues

Bash completion

Hello there

when I was being lazy and pressing tab to list the files, this happened:

$ wtf <TAB>
cut: /usr/share/misc/acronyms*: No such file or directory
-f

Even if I remove wtf from my PATH and try again, that happens. Could it be because of wtf(6)? But I don't think it is installed..

support a dry run type option

I found it useful to run find, find many files, then do wtf -o /dev/null to preview the changes, then wtf -i if I'm satisfied. There should be a built in option which acts like -o /dev/null.

Add option for formfeed characters

Currently, they are deleted by default as whitespace at the end of the line.
I think they should not be removed by default, and maybe an option
added to remove them. They are part of many gnu programs that
have very very widespread use:
https://www.gnu.org/prep/standards/standards.html#index-formfeed
These characters are also known as ctrl-L ^L, ascii character #12

git-commit hook careful version throwing errors (and not converting tabs to spaces)

D. Lenski,

I switched today to the "careful version" since it seems more robust and may be better if i do more advanced git things in the future. My entire script is:

#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".

if git rev-parse --verify HEAD >/dev/null 2>&1
then
    against=HEAD
else
    # Initial commit: diff against an empty tree object
    against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --bool hooks.allownonascii)

# Redirect output to stderr.
# exec 1>&2

# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
    # Note that the use of brackets around a tr range is ok here, (it's
    # even required, for portability to Solaris 10's /usr/bin/tr), since
    # the square bracket bytes happen to fall in the designated range.
    test $(git diff --cached --name-only --diff-filter=A -z $against |
      LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
    cat <<\EOF
Error: Attempt to add a non-ASCII file name.

This can cause problems if you want to work with people on other platforms.

To be portable it is advisable to rename the file.

If you know what you are doing you can disable this check using:

  git config hooks.allownonascii true
EOF
    exit 1
fi

# Use WTF to make consistent whitespacing. (see https://github.com/dlenski/wtf for usage)
wtf_options='--change-tabs 4 --tab-space-mix'
tmp=$(mktemp)
# Run Whitespace Total Fixer on index contents WITHOUT touching working directory
git diff-index --cached HEAD --diff-filter=ACMRTU |
while read _ MODE _ SHA1 _ FILE
do
    if ! ( git cat-file blob $SHA1 | wtf $wtf_options > $tmp ); then
        git update-index --cacheinfo $MODE $(git hash-object -w $tmp) "$FILE"
        echo "Fixed whitespace in $FILE" >&2
    fi
done
rm -f $tmp 2> /dev/null

# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

I used the options I wanted '--change-tabs 4 --tab-space-mix', but I excluded -i because i figured you're dealing with it a different way.

The error is (i'm using zsh, but it should be running in sh anyway):

~/programming/knewwaves % git commit -m "test WTF 2"
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix 
.git/hooks/pre-commit: line 55: $tmp: ambiguous redirect
error: option 'cacheinfo' expects <mode>,<sha1>,<path>
usage: git update-index [<options>] [--] [<file>...]

    -q                    continue refresh even when index needs update
    --ignore-submodules   refresh: ignore submodules
    --add                 do not ignore new files
    --replace             let files replace directories and vice-versa
    --remove              notice files missing from worktree
    --unmerged            refresh even if index contains unmerged entries
    --refresh             refresh stat information
    --really-refresh      like --refresh, but ignore assume-unchanged setting
    --cacheinfo <mode>,<object>,<path>
                          add the specified entry to the index
    --chmod (+/-)x        override the executable bit of the listed files
    --assume-unchanged    mark files as "not changing"
    --no-assume-unchanged
                          clear assumed-unchanged bit
    --skip-worktree       mark files as "index-only"
    --no-skip-worktree    clear skip-worktree bit
    --info-only           add to index only; do not add content to object database
    --force-remove        remove named paths even if present in worktree
    -z                    with --stdin: input lines are terminated by null bytes
    --stdin               read list of paths to be updated from standard input
    --index-info          add entries from standard input to the index
    --unresolve           repopulate stages #2 and #3 for the listed paths
    -g, --again           only update entries that differ from HEAD
    --ignore-missing      ignore files missing from worktree
    --verbose             report actions to standard output
    --clear-resolve-undo  (for porcelains) forget saved unresolved conflicts
    --index-version <n>   write index in this format
    --split-index         enable or disable split index

Fixed whitespace in docs/source/cc_content_style_guide.rst
[master f765cba] test WTF 2
 1 file changed, 1 insertion(+), 1 deletion(-)
~/programming/knewwaves % 

I'm guessing you either meant to create a temporary directory instead of a temporary file, or you meant to do something like mktemp /tmp/WTF_output.XXXXX instead of just mktemp. Or perhaps mktemp behaves differently in OSX or specifically

~/Desktop/test % sh --version
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.

than on your system?

Would be great to have a robust version that works for everyone. Then we can delete the "simple version" entirely.

Question: how to check for non-mixed tab or space indentation with wtf ?

Is there a way to make wtf check for specific non-mixed tab or space indentation ?

man page has wrong file name

The man page is named "gp-saml-gui.8" (using hypens as separators)

But recently the command has switched name to "gp_saml_gui.py" (using underscores as separators)

Request: no-mixed-spaces mode

Please see attached foo.txt. Line 2 has fifteen leading spaces. Running

wtf -v -s -y 8 foo.txt | xxd -g1

changes the first 8 spaces to a tab and leaves the last 7 spaces:

	CHANGED 8 spaces to tabs on 1 lines
00000000: 30 31 32 33 34 35 36 37 38 39 61 62 63 64 65 66  0123456789abcdef
00000010: 0a 09 20 20 20 20 20 20 20 68 65 6c 6c 6f 21 0a  ..       hello!.

I would like to request a mode in which any trailing group of spaces after a leading group of tabs would be converted to a single tab. Unlike -y N, this would convert every group of up to N spaces to a tab.

The use case is Linux kernel coding, which uses hard tabs. I care more about consistency than exact-column indentation. clang-format will give me tab+space mix, and I'd like to use wtf to convert those last few spaces to one additional tab per line.

Thanks for considering this request!

Need testers for Python 3 version

It's 2020, and Python 2.7 is at EOL.

Please test the python3 branch and give feedback on anything that doesn't work properly. (So far, so good for me 🤞)

Request new mode: fix based on .editorconfig

I use wtf a great deal, and have also started using .editorconfig files in the last year or so. A .editorconfig file specifies whitespace type, indent size, and other editor settings for the files in a repo.

Would you be willing to add (or accept a PR for, if I can find the time :D ) a mode that reads the .editorconfig file in order to find the settings to apply? For me, that would mean I would no longer have to specify the specific trailing/eof/eol/tab-space command-line options, and would always get consistent results.

Python code to load the .editorconfig files is already available. Thanks for considering this request!

Add option to report full crlf file

wtf has option to convert to crlf, but not report a full crlf file as a problem, only a mix of crlf and lf. git can (and looks like by default, but I haven't tested) report a full crlf file as a whitespace problem.

Add Space to Tab Conversion

Any plans on adding "Space to tab conversion" as mentioned in the todo section?

It would be nice for wtf.py to also be able to convert leading N spaces to tabs.

respect gitattributes

When I commit a PDF file, it runs wtf on the PDF (I removed the -q switch in the hook):

> git add file.pdf
> git commit -m "test"
<stdin>:
	CHOPPED 9 lines with trailing space
	CHOPPED 0 blank lines at EOF
	no change to newline at EOF
	CHANGED 16 line endings which didn't match lf from first line
	WARNED ABOUT 0 lines with mixed tabs/spaces

This can mess up the PDF. I understand that I can bypass the filter with -n, but wouldn't it be better for wtf to respect files explicitly declared as binary in .gitattributes?

Many thanks,

Jean-Luc

Add fix option for spaces followed by tabs in whitespace at beginning of line

Having this option would make wtf contain every whitespace fix that git has + more. My git (1.7.9.5) converts these spaces into a tab by default when fixing whitespace.

Change `python` to `python3`

wtf/wtf.py

Line 1 in fe5618a

#!/usr/bin/env python

Some distributions such as Ubuntu 22.04 have only python3 installed by default, no python. Python 3 should be available on (almost) all developer machine nowadays:

$ wtf.py 
/usr/bin/env: 'python': No such file or directory
$