recursive_directory_iterator::increment fails assertion on corrupted file system

[boostorg]

recursive_directory_iterator::increment fails assertion on corrupted file system

[Lastique]

I haven't looked into the issue, but since you're working on it, did you consider what the C++ Standard says about it? Out of all possible solutions, I think the standard behavior would be more preferable.

[xRodney]

I have been able find some information about the other issue I reported (#112) in the draft of the standard (http://eel.is/c++draft/fs.class.rec.dir.itr#fs.rec.dir.itr.members)

I have found this https://en.cppreference.com/w/cpp/filesystem/recursive_directory_iterator

If the recursive_directory_iterator reports an error or is advanced past the last directory entry of the top-level directory, it becomes equal to the default-constructed iterator, also known as the end iterator.

This is not in the standard (at least in the draft I referred to) and I would find this behaviour unfortunate as it prevents the user to recover from the error in any way.

[xRodney]

Speaking of recovery, I have suggested two separate means of recovery in my PR #114.

The differentiate between where the error occurs:

• If opendir raises an error (such as Permission denied), the recovery is to disable recursion to this directory
• If readdir raises an error (such as Input/output error or Result too large on Apple), the recovery involves popping that directory and then disabling recursion back there

That means that the caller has little control over the recovery, because from the error_code it may not be evident which function failed, and thus they do not know if they should pop the current dir or not

[Lastique]

I think the current standard draft (N4820) is pretty clear on the intended behavior, and I think that behavior is sane. Specifically:

• If directory_options::skip_permission_denied option was specified on iterator construction and an access denied condition was detected (which happens on opendir) then that subdirectory is skipped as if it's empty.
• On any other errors indicate failure (in case of increment - throw an exception).

In the latter case the user can try to recover by manually calling depth and pop on the iterator. It is not the iterator's job to attempt to conceal or recover from errors such as this. Side effects like returning the same directory twice are not acceptable. In other words, readdir always means an exception, this part is not going to change.

Currently, there is no directory_options in Boost.Filesystem, symlink_option is used instead and it doesn't have an equivalent of skip_permission_denied. I think, we should add directory_options, including the skip_permission_denied, and deprecate symlink_option.

[Lastique]

In the latter case the user can try to recover by manually calling depth and pop on the iterator.

Upon reading the standard further it seems like this recovery is not allowed since the iterator is required to become equal to the end iterator on error. This follows from [fs.class.rec.dir.itr]/3 and [fs.class.directory.iterator]/3.

[Lastique]

See if the new option pop_on_error from 9a14c37 suits your needs.

[nishaparakkat]

Hi, I took latest boost and filesystem submodule shows latest commit as f795041.
The fix in 9a14c37 is already present in the git tree.

I tried to check if the reproduction case mentioned in the above comments using fusepy is fixed.
But I get the below error

my-ThinkStation-P910:/myhome/mytest/boost/run_bugboost_171/from_site_113$ sudo -s
root@my-ThinkStation-P910:/myhome/mytest/boost/run_bugboost_171/from_site_113# python3 memoryfs.py mountpoint/ &
[1] 1136
root@my-ThinkStation-P910:/myhome/mytest/boost/run_bugboost_171/from_site_113# mkdir -p mountpoint/dirxx/dir2
root@my-ThinkStation-P910:/myhome/mytest/boost/run_bugboost_171/from_site_113# mkdir -p mountpoint/diryy/dir2
root@my-ThinkStation-P910:/myhome/mytest/boost/run_bugboost_171/from_site_113# ./recursive_ls_171

usage: recursive_ls [path]

In directory: "/myhome/mytest/boost/run_bugboost_171/from_site_113"

"/myhome/mytest/boost/run_bugboost_171/from_site_113/recursive_ls.cpp"
"/myhome/mytest/boost/run_bugboost_171/from_site_113/bugboost_171_stdc++11"
"/myhome/mytest/boost/run_bugboost_171/from_site_113/mountpoint" [directory]
"/myhome/mytest/boost/run_bugboost_171/from_site_113/mountpoint/diryy" [directory]
recursive_ls_171: /myhome/mytest/boost/boostgit/boost_lib_1.71/include/boost/filesystem/directory.hpp:556: boost::filesystem::directory_entry& boost::filesystem::recursive_directory_iterator::dereference() const: Assertion `(!is_end())&&("dereference of end recursive_directory_iterator")' failed.
Aborted (core dumped)

I built the recursive_ls using below :
g++ -std=c++17 -Os -Wall -pedantic recursive_ls.cpp -lboost_system -lboost_filesystem -lpthread -lboost_thread -L/myhome/mytest/boost/run_bugboost_171/from_site_113boost_lib_1.71/lib -L /usr/lib -I/myhome/mytest/boost/run_bugboost_171/from_site_113/boost_lib_1.71/include/ -o run_bugboost_171_17

tried a version with -std=c++11 too but i get the same error.

Is this issue fixed ? or am i missing something here?

[nishaparakkat]

#113 #hi all,
if I change the recursive_ls.cpp that was used for reproducing the issue, so that

 for (fs::recursive_directory_iterator dir_itr(p);
          dir_itr != end_iter;)

becomes

for (fs::recursive_directory_iterator dir_itr(p);
          dir_itr != end_iter;++dir_itr)

and I remove the lines

boost::system::error_code ec;
dir_itr.increment(ec);

Then I get the below error.

usage: recursive_ls [path]

In directory: "/myhome/mytest/boost/run_bugboost_171/from_site_113"

"/myhome/mytest/boost/run_bugboost_171/from_site_113/recursive_ls.cpp"
"/myhome/mytest/boost/run_bugboost_171/from_site_113/run_bugboost_171_17_no_except"
"/myhome/mytest/boost/run_bugboost_171/from_site_113/bugboost_171_original"
"/myhome/mytest/boost/run_bugboost_171/from_site_113/mountpoint" [directory]
terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
what(): filesystem::recursive_directory_iterator increment error: Input/output error
Aborted (core dumped)

This issue is seen for our software that uses a boost version 1.65 too.
Is this a known issue?

[Lastique]

The test code in the OP is incorrect because it dereferences the iterator in std::cout lines after an operation failed on the iterator. The iterator becomes singular (i.e. equal to end) on error.

Is this a known issue?

I'm not a Python expert, but I think memoryfs.py is intended to simulate a corrupted filesystem. Thus the failure on iteration.