cfv-project/cfv

Command-line File Verify

cfv's Issues

Add Windows and macOS runners to CI pipeline

We also want to support Windows and macOS. Therefore, it would be useful to have Windows and macOS runners in this CI pipeline: https://github.com/cfv-project/cfv/blob/python3/.github/workflows/ci-python3.yml

Update README

The README should contain updated install instructions and requirements.

Move man page to /usr/share/man

cfv seems to ship manpages to /usr/man/man1, following a really old standard from 1995, which has already been superseded in 1997. The current standard suggests to keep man pages in /usr/share/man. cfv should follow this standard.

Module 'TERMIOS' not found

Instead, it should be renamed to 'termios' (like two lines above):

cfv/lib/cfv/term.py

Line 13 in 1debfeb

from TERMIOS import TIOCGWINSZ

Re-format source code

Re-format the source code for better readability.
I would suggest to use the PEP-8 style guide (https://www.python.org/dev/peps/pep-0008/).
We could use flake8 as checking tool (https://pypi.org/project/flake8/).

Add macOS support

Currently, there are a few issues, why cfv including the test suite does not run on macOS. We have to investigate the issues and fix them.

I consider shifting back macOS support to a later release. If anyone wants to use cfv on macOS, please speak up.

Python 3 Support

Python 2 will be end-of-life in 2020 (https://www.python.org/dev/peps/pep-0373/#update), so we have to add support for Python 3.

Investigate and fix deprecation warnings

Some deprecated modules and functions are used. We have to investigate those and move them to a proper supported module/function.

Python provides the -W error argument to ease finding such issues. For example:

$ python3 -W error test/test.py
Traceback (most recent call last):
  File "test/test.py", line 48, in <module>
    import cfvtest
  File "/[...]/cfv/test/cfvtest.py", line 26, in <module>
    import imp
  File "/usr/lib/python3.6/imp.py", line 33, in <module>
    DeprecationWarning, stacklevel=2)
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

Once fixed, we should also add these Python flags in our CI pipeline.

Static typing with e.g. mypy

Hello,

I noticed cfv does not use type hints. Type hints and type checkers such as mypy are great aids for making robust code. Strongly recommended.

Cheers,
J

Add Windows support

Currently, there are a few issues, why cfv including the test suite does not run on Windows. We have to investigate the issues and fix them.

Add support for BLAKE3 (b3sum files)

Hello,

Big thanks for reviving the cfv project! I just installed it and it works great Have you considered adding Blake 3 support? It is a very fast hash algo.

My other feature request is an option to update the hash only if the file modify time is later than then checksum file.

Cheers,

J

Distribute via Debian

Update the Debian packaging to fix outstanding issues that caused the package to be removed from Debian Buster (https://tracker.debian.org/pkg/cfv).

Test suite expects `/data1` to not exists

As reported by @Terry-Kennedy, a test case fails, when /data1 exists: #8 (comment)

>>> failed test: /sysprog/terry/cfv-python3/test/cfv -ZNVRMUI --unquote=no --fixpaths="" --strippaths=0 --showpaths=autorelative --progress=no --announceurl=url --noprivate_torrent --strippaths=none -T -f teststrip-none.csv4 
/data1 : file size does not match (13!=33)
teststrip-none.csv4: 1 files, 0 OK, 1 badsize.  0.000 seconds, 0.0K/s
FAILED (4)
  File "test/test.py", line 1876, in <module>
  File "test/test.py", line 1719, in all_tests
  File "test/test.py", line 269, in test_generic
  File "test/test.py", line 254, in test_log_results
  File "test/test.py", line 237, in test_log_finish

It is bad practice that the test suite makes assumptions about things outside it's own scope (i.e. temp directory). However, the test case cannot easily be adjusted without modifying it's intention. Therefore, we just change the expected test outcome, depending on if the /data1 exists or not. Leaving only an unexpected error, if someone has a file /data1 with exactly the same content as https://github.com/cfv-project/cfv/blob/python3/test/testdata/data1.

DeprecationWarning: the imp module is deprecated in favour of importlib

HI,
running tests, I noticed this warning:

cfv/test/cfvtest.py:26: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

I didn't file a PR as I'm not exactly sure what

cfv/test/cfvtest.py

Line 205 in 08ee267

imp.load_source('cfvwrapper', cfvfn, f)

is used for.

Bash cfv tool

Hello,

I haven't been able to run the Python 2.7 based cfv for years on a stock Ubuntu due to some dependencies. As a long-time user of (Python-based) cfv, I decided to write a Bash script to do roughly the same thing and with less dependencies to "often" (once a decade) changing libs, modules. Just in case someone is interested.

Bash bcfv

"Tested and works on Linux"

Windows no context switch to ignore symbolic and hard links

if hashing recurive directories in windows there is no contect argument to not follow symbolic and hard linkss.

Add support for GNU coreutils b2sum files

Implemented in GNU coreutils with this commit: coreutils/coreutils@ea94589

Refactor Test Suite for use in CI

The most severe issues are already resolved.
However, it would be nice to be able to run unit tests, short CLI tests and long CLI tests separately.

Fix flake errors in python3 branch

Similar to #16 the flake errors should also be fixed in the python3 branch. Furthermore, the flake test should also be marked fatal.

Fix flake8 issues

Flake8 is currently running in non-fatal mode in our CI because of some issues.
We should fix the issues and make flake8 fail our pipeline again.

$ flake8 --exclude=build,venv --ignore= --max-line-length=200 --max-complexity=75 --show-source --statistics .
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/2.7.18/x64
./lib/cfv/osutil.py:112:78: E741 ambiguous variable name 'I'
def strippath(filename, num='a', _splitdrivere=re.compile(r'[a-z]:[/\\]', re.I)):
                                                                             ^
./lib/cfv/common.py:738:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:900:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1400:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1505:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1537:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1572:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1649:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/strutil.py:6:14: E741 ambiguous variable name 'l'
def safesort(l):
             ^
./test/cfvtest.py:49:26: E741 ambiguous variable name 'l'
    def writelines(self, l):
                         ^
10    E741 ambiguous variable name 'l'
Error: Process completed with exit code 1.

Fix clipping of max open files in CI pipeline

In the CI pipeline the maximum number of open file descriptors is clipped to 4096 for the manyfiles_test:

max open files is big (65536) clipping to 4096.  Use --long to try the real value

Therefore, the manyfiles_test cannot really detect issues with leaking file descriptors. We should reduce the maximum number of open file descriptors limit to <=4096 before running the tests in the CI pipeline. We can do this with ulimit -n 4096.

Distribute via PyPI

Future releases should be available via the Python Package Index (https://pypi.org/).

Issue warnings when external verification tools are not installed

The test suite can verify created files using external tools like cksfv and *sum. However, if they are not installed the verification is just silently skipped. We should at least issue a warning, if the tools are not installed.

Distribute as AUR package

Thanks for Python3 support! Quickly made an AUR package here: https://aur.archlinux.org/packages/cfv-git

Flake fails

Running the code style checker flake as expected by the CI script currently causes some errors:

$ flake8 --exclude=build,venv --ignore= --max-line-length=200 --max-complexity=75 --show-source --statistics .
./lib/cfv/common.py:738:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:900:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1400:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1505:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1537:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1572:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/common.py:1649:34: E741 ambiguous variable name 'l'
    def do_test_chksumline(self, l):
                                 ^
./lib/cfv/osutil.py:112:78: E741 ambiguous variable name 'I'
def strippath(filename, num='a', _splitdrivere=re.compile(r'[a-z]:[/\\]', re.I)):
                                                                             ^
./lib/cfv/strutil.py:6:14: E741 ambiguous variable name 'l'
def safesort(l):
             ^
./test/cfvtest.py:49:26: E741 ambiguous variable name 'l'
    def writelines(self, l):
                         ^
10    E741 ambiguous variable name 'l'

Investigate and fix `bytes` to `str` conversation using `str()`

There still seem to be a lot of uses of str() to convert from bytes to str. We have to investigate those and move them to a proper decoding using something like .decode(correct_encoding).

Python provides the -bb argument to ease finding such issues. For example:

$ python3 -bb test/test.py --exit-early
>>> testing...
...........................Traceback (most recent call last):
  File "test/test.py", line 1874, in <module>
    failed += all_tests()
  File "test/test.py", line 1620, in all_tests
    ren_test(fmt)
  File "test/test.py", line 913, in ren_test
    flscmp(b'1', 0, flsf)
  File "test/test.py", line 897, in flscmp
    test_log_results('cmp %s for %s' % (fn, t), r, o, r, None)
BytesWarning: str() on a bytes instance

Once fixed, we should also add these Python flags in our CI pipeline.

Add FreeBSD runner to CI pipeline

@Terry-Kennedy tested cfv on FreeBSD and created a FreeBSD port (#8 (comment)). However, currently we do not have a CI runner for FreeBSD. To ensure support in the long run, we should have our CI pipeline also regularly run on FreeBSD.

Sadly, Github Actions does not directly support FreeBSD. One possible workaround would be using this action running a FreeBSD VM on the macOS runner: https://github.com/marketplace/actions/freebsd-vm