rbtcollins/fs_at

Filesystem 'at' implementations for Unix and Windows

*_at syscalls for Rust (*nix and Windows)

The Rust standard library does not (yet) offer at-style filesystem calls as a core feature. For instance mkdirat. These calls are essential for writing race-free filesystem code, since otherwise the state of the filesystem path that operations are executed against can change silently, leading to TOC-TOU race conditions. For Unix these calls are readily available in the libc crate, but for Windows some more plumbing is needed. This crate provides a unified Rust-y and safe interface to these calls.

Not all platforms behave identically in their underlying syscalls, and this crate doesn't abstract over fundamental differences, but it does attempt to provide consistent errors for key scenarios. As a concrete example creating a directory at the path of an existing link with follow disabled errors with AlreadyExists.

On Linux this is achieved by reading back the path that was requested, as atomic mkdir isn't yet available. mkdirat is used so the parent directory is reliable, but the presence of a link pointing to another part of the file system cannot be precluded.

On Windows this same scenario will either result in fs_at receiving a NotADirectory error from NtCreateFile, or the open succeeding but a race-free detection of the presence of the link is done using DeviceIoControl. Both cases are reported as AlreadyExists. The two codepaths exist because on Windows symlinks can themselves be files or directories, and the kernel type-checks some operations such as creating a directory or truncating a file at both the link target and the link source.

Truncate+nofollow also varies by platform: See OpenOptions::truncate.

MSRV policy

I'll keep this compiling against older rusts as long as it is easy, but not at the expense of a lot of code golf, or past CVEs in old releases of dependencies. Currently MSRV is 1.63. If there is a lot of interest in older versions I'm open to patches.

Usage

See the crate docs. But in short: use fs_at::OpenOptions, similar to std::fs::OpenOptions.

vs other crates

openat

openat is a nice wrapper around the Unix *at facilities. It doesn't offer Windows support, and it also requires adoption of a new Dir struct which owns the fd - which adds friction for interop with the rest of std.

cap_std

cap_std is a lovely rethink of many system interactions as operations on capabilities. Even more than openat, it steps away from the familiar std APIs and instead provides its own comprehensive ecosystem.

Unfortunately that doesn't use the full capabilities of the underlying OS - it layers on top of Rust's own IO stack in some cases (e.g. Windows, some non-Linux), leading to TOCTOU concerns. That is obviously fixable over time - if you want a high level API that will make insecure usage hard, I think cap-std is perfect.

The goal of fs_at isn't to reframe how we do IO though - but just to surface these important calls in an ergonomic way. Perhaps cap_std could layer on fs_at when it is finished.

Contributing

PR's as normal on Github.

Coverage - consider grcov.

export RUSTFLAGS="-Cinstrument-coverage"
export LLVM_PROFILE_FILE="fs-at-%p-%m.profraw"
cargo test && grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./target/debug/cove
rage/

Code of conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

fs_at's Issues

nightly builds failing in `proc-macro2` 1.0.56 crate

This is not a direct dependency.

$ cargo tree -i proc-macro2
proc-macro2 v1.0.56
├── quote v1.0.26
│   ├── syn v1.0.109
│   │   └── test-log v0.2.11 (proc-macro)
│   │       [dev-dependencies]
│   │       └── fs_at v0.1.9 (C:\Users\robertc\Documents\src\fs_at)
│   └── test-log v0.2.11 (proc-macro) (*)
├── syn v1.0.109 (*)
└── test-log v0.2.11 (proc-macro) (*)

So this doesn't affect users on nightly, but we can't test on nightly at the moment.

  Running `/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name proc_macro2 --edition=2018 /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.56/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=516e696159394256 -C extra-filename=-516e696159394256 --out-dir /home/runner/work/fs_at/fs_at/target/debug/deps -L dependency=/home/runner/work/fs_at/fs_at/target/debug/deps --extern unicode_ident=/home/runner/work/fs_at/fs_at/target/debug/deps/libunicode_ident-3357ee1867c8a168.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
error[E0635]: unknown feature `proc_macro_span_shrink`
  --> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.56/src/lib.rs:92:30
   |
92 |     feature(proc_macro_span, proc_macro_span_shrink)
   |                              ^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0635`.
error: could not compile `proc-macro2` (lib) due to previous error

Caused by:

mkdir_at on Unix is complicated

See the code but in short we're working around race conditions such as privileged access target control via symlinks; while there aren't any TOCTOU's present, there is the possibility for creating a directory in an arbitrary path (which mkdir itself has and there is resistance to fixing in the Linux kernel community).

At this point there doesn't seem to be a better path, but should one arise we should use it.

Crate does not compile on iOS

When trying to build the create on iOS, the build fails with the error:

   Compiling fs_at v0.1.7
error[E0425]: cannot find value `O_PATH` in crate `libc`
   --> /Users/felix.barz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fs_at-0.1.7/src/unix.rs:139:55
    |
139 |             libc::O_RDONLY | libc::O_NOFOLLOW | libc::O_PATH | libc::O_CLOEXEC | libc::O_NOCTTY;
    |                                                       ^^^^^^ not found in `libc`

For more information about this error, try `rustc --explain E0425`.
error: could not compile `fs_at` (lib) due to previous error

When checking the sources, whe can see that open_path_at is disabled for macos: https://github.com/rbtcollins/fs_at/blob/main/src/unix.rs#L136

This should also be disable for iOS. When adding target_os = "ios" to the exclusions (including a few other places), the library compiles again.

Return file type from readdir

Continuing the thread from #12 and sourcefrog/conserve#174.

In a tree that might have symlinks I'd want an option to open files with O_NOFOLLOW on Linux. (It's in Nix and libc; it doesn't seem to be in std?) (I filed #14.)

If the entry is a symlink, that will fail with ELOOP. I could then try to read it with readlinkat. If it then changes back to a file, readlinkat will give you an error.

I don't think I see the harm in DirEntry giving you an indication of the file type so that you know where to begin. There is a chance of a race, that when you read the dir it's a symlink and then it changes to be a file, or vice versa. But the worst that happens is that the call you use to read the file or link will fail. So I don't think the race is meaningful?

A different approach would be to begin with open O_PATH on Linux, and then you can stat it, find out what type of thing it is, and then proceed to either "really open it" or read the link. In this case you only pass the filename once, which seems nice.

O_PATH seems to be Linux specific https://stackoverflow.com/q/61337371.

This also seems to need more syscalls. The arguably straightforward way is

1. getdents gives you the name and tells you it's a real file
2. open the file (with O_NOFOLLOW)
3. (do stuff then) close

If readdir does not tell you the type, on Linux, you can do

1. getdents
2. openat O_PATH
3. fstatat tells you what you opened is a file
4. openat "" on that path fd to actually open the file
5. close the O_PATH fd
6. (do stuff then) close the second fd

This just seems like more steps though.

On non-Linux Unix... actually I don't see how you can do anything other than attempt to open it or readlink it? But you might as well have a clue which of them is going to work?

Build failed for `x86_64-unknown-illumos` and `x86_64-unknown-netbsd`

Got an error:

error[E0432]: unresolved import `libc::openat64`
  --> /cargo/registry/src/github.com-1ecc6299db9ec823/fs_at-0.1.1/src/unix.rs:17:13
   |
17 |         use libc::openat64;
   |             ^^^^^^--------
   |             |     |
   |             |     help: a similar name exists in the module: `openat`
   |             no `openat64` in the root

For more information about this error, try `rustc --explain E0432`.
error: could not compile `fs_at` due to previous error

I guess it similar with #55. See more at: https://github.com/rust-lang/rustup/actions/runs/4397256299/jobs/7700257156

Support additional file objects - hardlinks

posix semantics on windows

We should use posix semantics to delete on windows.

Rust platform support claims windows 7 and up.

I can't find any reference to the necessary structure to request this the way the stdlib does in the MS docs. The closest I've come just notes its existence, not when APIs started accepting it, nor what will happen on older Windows versions if the structure class, or bits in the flag that were defined more recently are passed in.

@ChrisDenton sorry for tagging you out of the blue but I'm hoping you can give a little guidance / point to docs about the thing to do here.

My inclination was to first version-check the runtime OS for windows 10, and then and only then attempt the new information class, falling back on error to the older class, but the Rust stdlib seems to unconditionally use the new information class, so if thats safe it seems more straightforward...

Trying to build for target_os="ios" fails

fs_at/src/unix.rs

Line 17 in fba9c86

use libc::openat64;

When using this crate on a project built for target_os="ios" the above line is causing a build failure.

error[E0432]: unresolved import `libc::openat64`
  -->~/.cargo/registry/src/github.com-1ecc6299db9ec823/fs_at-0.1.1/src/unix.rs:17:13
   |
17 |         use libc::openat64;
   |             ^^^^^^--------
   |             |     |
   |             |     help: a similar name exists in the module: `openat`
   |             no `openat64` in the root

Remove things that should be in windows-sys

#40 was solved by reimplementing things that windows-sys doesn't yet have. Tracking bug for burning that down. See links to github in the source to see what is pending.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update Rust crate env_logger to 0.11.0
• Update Rust crate nix to 0.27.1
• Update Rust crate windows-sys to 0.52.0
• Update actions/cache action to v4
• Update actions/checkout action to v4
• Lock file maintenance
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update Rust crate cvt to 0.1.2
• Update Rust crate log to 0.4.20

Detected dependencies

cargo

Cargo.toml

• cfg-if 1.0.0
• cvt 0.1.1
• log 0.4.17
• env_logger 0.10.1
• fs-set-times 0.20.1
• rayon 1.8.1
• tempfile 3.9.0
• test-log 0.2.14
• libc 0.2.152
• nix 0.26.2
• aligned 0.4.1
• once_cell 1.19.0
• windows-sys 0.48.0

github-actions

.github/workflows/publish.yaml

• actions/cache v3

.github/workflows/rust.yml

• actions/checkout v3
• Swatinem/rust-cache v2
• actions/checkout v3
• Swatinem/rust-cache v2
• actions/checkout v3
• Swatinem/rust-cache v2

---

• Check this box to trigger a request for Renovate to run again on this repository

Migrate to windows[-sys] crate

This is partially done, but a little stuck on:

• SECURITY_CONTEXT_TRACKING_MODE (no hits in https://microsoft.github.io/windows-docs-rs/doc/windows/?search=SECURITY_CONTEXT_TRACKING_MODE )
• NT_SUCCESS (likewise). Could reimplement obviously
• RtlInitUnicodeStringEx (not RtlInitUnicodeString which is supported)
• A bunch of stuff from ntapi::ntioapi like SYMLINK_FLAG_RELATIVE, REPARSE_DATA_BUFFER, ...

@Xaeroxe offered in XAMPPRocky/remove_dir_all#42 to help out; this is a call for that help :)

enh: hardlink support

https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-ntsetinformationfile windows - FileLinkInformation can link to an existing file

RootDirectory permits safe construction of the link object itself, and the existing file handle is provided in the call to NtSetInformationFile, so this is safe.

For unix, linkat takes two dir handles, so similarly safe.

Support ReadDir relative to an fd?

For https://github.com/sourcefrog/conserve to use this very much I think it would want a way to open and read directories relative to a different directory fd.

This might already be possible but I don't think I see it?

On Linux this would be something like openat then getdents64? Or on more generic Unix perhaps openat, fdopendir, readdir?

Support additional file objects - symlinks

Build fails on NetBSD

Hi,

Any chance to get this fixed?
denisidoro/navi#830

Thanks!

Option to open files with O_NOFOLLOW

Consider ManuallyDrop to avoid duping dir handles

Example: https://github.com/rust-lang/rust/blob/df7fd9995f10627f25ccb325f693a11b3395a73c/library/std/src/sys/windows/fs.rs#LL1118C33-L1118C50