danburkert/memmap-rs

cross-platform Rust API for memory mapped IO

memmap

A Rust library for cross-platform memory mapped IO.

Features

• file-backed memory maps
• anonymous memory maps
• synchronous and asynchronous flushing
• copy-on-write memory maps
• read-only memory maps
• stack support (MAP_STACK on unix)
• executable memory maps
• huge page support

Platforms

memmap should work on any platform supported by libc. memmap requires Rust stable 1.13 or greater.

memmap is continuously tested on:

• x86_64-unknown-linux-gnu (Linux)
• i686-unknown-linux-gnu
• x86_64-unknown-linux-musl (Linux MUSL)
• x86_64-apple-darwin (OSX)
• i686-apple-darwin
• x86_64-pc-windows-msvc (Windows)
• i686-pc-windows-msvc
• x86_64-pc-windows-gnu
• i686-pc-windows-gnu

memmap is continuously cross-compiled against:

• arm-linux-androideabi (Android)
• aarch64-unknown-linux-gnu (ARM)
• arm-unknown-linux-gnueabihf
• mips-unknown-linux-gnu (MIPS)
• x86_64-apple-ios (iOS)
• i686-apple-ios

License

memmap is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2015 Dan Burkert.

memmap-rs's Issues

Support madvise.

It might be useful to add support for POSIX madvise, and/or something like CreateFile with FILE_FLAG_SEQUENTIAL_SCAN on Windows.

Is this something within the scope of this crate? Or is this be something to be left to another crate?

Mmap fails for empty files on Windows

As per the MSDN documentation for CreateFileMapping:
An attempt to map a file with a length of 0 (zero) fails with an error code of ERROR_FILE_INVALID. Applications should test for files with a length of 0 (zero) and reject those files.
I believe this should not be the expected behavior for the memmap-rs crate, and we should check if the size is 0 (we already get the size before mapping the file anyway), and return an empty-slice backed Mmap struct in that case.

Safety of Mmap::as_slice ?

The documentation for this method includes:

Unsafety

The caller must ensure that the file is not concurrently modified.

But since the filesystem is shared with other processes that might do anything, this seems very difficult to ensure. (I imagine that an application could be run in a container like Docker to give it a private filesystem?)

So what’s the worst that could happen? https://stackoverflow.com/questions/21286870/how-safe-are-memory-mapped-files-for-reading-input-files seems a bit hand-wavy but suggests: not much.

I’m considering using this method to read (hopefully more efficiently than with File::read) files that are usually not modified, but they might be modified for example when the system’s package manager updates them to a new version.

I do not mind if reading a byte at the same location twice gives different values, or if reading two locations give inconsistent values (because a write has happened in between the two reads). This might cause my program to unexpectedly return Err or (safely) panic, but that’s ok.

I do mind if this is Undefined Behavior of the sort that can cause anything to happen, including potentially being exploited for remote code execution or other fun stuff.

If only the former can happen, should this method really be unsafe?

Support `Mmap` on platforms without real mmap support

It could just read a file to Vec<u8> and then deref it to &[u8] on dereferencing Mmap. This would make it easier to port programs using an immutable mmap for efficiency reasons to platforms without mmap support, like WebAssembly.

cc rust-lang/miri#722

Allow mapping memory safely using `volatile` crate?

It seems to me, reading through your justification for making file-backed memory maps unsafe, that the main reason for doing this is that they could change underneath you and thus violate Rust's memory model. However, a solution to this problem exists in the form of the volatile crate's Volatile<T> wrapper, which provides a safe way to read and write memory addresses which could change (with the added bonus that reads and writes are guaranteed not to be optimized away). It would still be beneficial to keep the existing unsafe interface in service, because the implementation of the volatile crate requires that data be copied whenever it is read, which may introduce an unacceptable amount of overhead for some users, but using it would provide other users the option of a safe interface.

Using the crate would just require the creation of SafeMmap and SafeMmapMut types, identical to their unsafe counterparts except that their Deref targets were [Volatile<u8>] rather than [u8].

https://linux.die.net/man/2/mremap

Implement StableDeref?

The Deref implementation of Mmap seems to fulfill the requirements of the StableDeref trait from https://crates.io/crates/stable_deref_trait, if I understand things correctly. So an implementation for it would be helpful for those who want to put an Mmap together with references into the mapped memory into a struct (or similar things). It will require an additional crate dependency, though (maybe opt-in or opt-out via feature flag?)

Put on crates.io

would be nice to see this on crates.io

Docs should say why file-backed maps are unsafe

In contrast to the functions to create anonymous maps, all the file-backed ones are marked unsafe. However, the docs don't say why this is.

What I can glean from #25 is that while memmap would have one believe that one gets an immutable &[u8] out of a map, in reality the file can change underneath and one may need to take extra measures like file locking and whatnot.

Is this accurate? If so, I can submit a PR to update the docs to that effect.

Please publish the newer version to crates.io

The current version on crates.io is failed to compile.

Code 87 when attempting to write files

Issue

The program can read files but switching to a mutable map in memmap-rs or memmap2 on Windows breaks due to an inappropriate call to Windows. The resulting error is a code 87 incorrect parameter error. I would love some guidance on what the actual issue is or a patch as I am actively trying to build a cache system after all other options appear to have moved to a NoSQL database due to popularity. My goal is to achieve a near-real-time tracking system for streaming ETL. Memory mapping allows me to avoid constantly loading files from memory in a SQL-lite memory-mapped style approach to storage.

Action

Switch from an immutable memory map to a mutable map.

Code in memmap/memmap2/memmap-rs

let mut file2 = File::open("E:\testfiles\mmap\test.mmap").unwrap();
let map = unsafe{MmapOptions::new().map(&file2).unwrap()};
let mut mut_map = map.make_mut().unwrap();

Expected Result

The retrieval of a writable/mutable memory mapping.

Actual Result

The action on line 3 fails with an apparent winapi issue:

thread 'main' panicked at 'called Result::unwrap() on an Err value: Os { code: 87, kind: Other, message: "The parameter is incorrect." }', src\main.rs:35:23

Stack Trace

thread 'main' panicked at 'called Result::unwrap() on an Err value: Os { code: 87, kind: Other, message: "The parameter is incorrect." }', src\main.rs:35:23
stack backtrace:
0: backtrace::backtrace::trace_unsynchronized
at C:\Users\VssAdministrator.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.46\src\backtrace\mod.rs:66
1: std::sys_common::backtrace::_print_fmt
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\sys_common\backtrace.rs:78
2: std::sys_common::backtrace::_print::{{impl}}::fmt
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\sys_common\backtrace.rs:59
3: core::fmt::write
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libcore\fmt\mod.rs:1076
4: std::io::Write::write_fmtstd::sys::windows::stdio::Stderr
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\io\mod.rs:1537
5: std::sys_common::backtrace::_print
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\sys_common\backtrace.rs:62
6: std::sys_common::backtrace::print
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\sys_common\backtrace.rs:49
7: std::panicking::default_hook::{{closure}}
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:198
8: std::panicking::default_hook
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:218
9: std::panicking::rust_panic_with_hook
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:486
10: std::panicking::begin_panic_handler
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:388
11: core::panicking::panic_fmt
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libcore\panicking.rs:101
12: core::option::expect_none_failed
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libcore\option.rs:1272
13: core::result::Result<memmap::MmapMut, std::io::error::Error>::unwrapmemmap::MmapMut,std::io::error::Error
at C:\Users\aseva.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\result.rs:1005
14: test_other_stuff::main
at .\src\main.rs:35
15: std::rt::lang_start::{{closure}}<()>
at C:\Users\aseva.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\rt.rs:67
16: std::rt::lang_start_internal::{{closure}}
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\rt.rs:52
17: std::panicking::try::do_call
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:297
18: std::panicking::try
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panicking.rs:274
19: std::panic::catch_unwind
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\panic.rs:394
20: std::rt::lang_start_internal
at /rustc/fe10f1a49f5ca46e57261b95f46f519523f418fe/src\libstd\rt.rs:51
21: std::rt::lang_start<()>
at C:\Users\aseva.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\rt.rs:67
22: main
23: invoke_main
at d:\agent_work\3\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
24: __scrt_common_main_seh
at d:\agent_work\3\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
25: BaseThreadInitThunk
26: RtlUserThreadStart
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
error: process didn't exit successfully: target\debug\test-other-stuff.exe (exit code: 101)

memmap doesn't support an execute-only mapping.

memmap doesn't support an execute-only mapping.

In particular, this is useful for JITs where the usage scenario is:

• allocate/mmap a buffer as ReadWrite
• write instructions into it
• then mark/mprotect the pages as execute only (Execute).

An article by Eli Bendersky: How to JIT - an introduction

Documentation / user guide

In my opinion, as someone who knows very little about the subject of memory-mapped files, quite a bit more documentation would be useful. This mostly isn't specific to this crate, but having the documentation in one place (ideally within the module documentation) would make the topic easier to learn.

• What are the main use-cases vs non-memory mapped files — performance, memory usage and behaviour differences with read-only and read-write usage. (E.g. is it useful when reading a file sequentially or only when seeking?)
• What are the main costs of mem-mapped files: presumably opening, reading a page, and flushing changes?
• Are there any guarantees when a memory-mapped file is read and modified by another process — e.g. if the writes are only appends, are reads safe? Is it OS-specific?
• Can mem-mapped files be locked by the OS? This may be a separate topic, but pointers welcome.
• How does paging work: entirely automatically in response to pointer dereferences? What's the page size (always 4kB?) & how should applications best allocate file memory to take advantage?

Use with DirectIO

Currently I'm using a mmap to modify a file opened with a O_DIRECT flag. This is so that I can write to the device in increments of the device's block size. However as I write more and more to the mmap it grows in memory. I'm not concerned about reading from the file/mmap, only writing. I only need to keep the last few bytes (to the lower block size bound) and what I am writing in the mmap.

For example, take this file on a device with a block size of 512 bytes.

|----------|----------|----------|------EOF
0         512       1024       1536

In this case I would want to the mmap offset to be 1536. Then I can write some data (24 bytes) to it, pad it 488 bytes of 0's and flush the 512 byte mmap. I suppose I would use std::fs::File::set_len to update the filesystem with the file's real length on every flush.

My question is is there a way to continuously truncate the start of the mmap, so it doesn't just keep growing and consuming memory as I write to it? Is mremap a good use for this and does memmap-rs provide any built in methods for this?

P.S. I'm new to using mmap and direct io so if I'm doing anything completely wrong, I apologize.

Support for msync

There's no reason to flush all outstanding changes in all cases. It might be nice if we supported msync

How to slice up mmap to share chunks of the same mmap?

I am working on a mmap-powered database driver, whose main struct looks like this:

pub struct WhisperFile<'a> {
    pub path: PathBuf,
    pub mmap: Mmap,
    pub header: Header,
    pub archives: Vec< Archive<'a> >,
}

I want the Archives to get their slice of the mmap property. But I can't figure out the rust lifetime gymnastics. Think it can be done? Seems safe if they're all tied to lifetime 'a. Any tips?

Add CI badges to Cargo.toml

Incorrectly uses usize instead of u64 for file offsets

As per the discussion here: https://users.rust-lang.org/t/rust-beginner-notes-questions/14928/81?u=peter_bertok

The file offset should be consistently u64 everywhere instead of usize, even on 32-bit platforms. In practice, I believe Windows doesn't actually support mapping more than 2 GB on 32-bit, so isize::MAX might be a better window-size limit.

Expand crate-level documentation

The root of a crate's documentation is the first thing a prospective user sees, and should provide an introduction to both the concepts the library deals with as well as quick examples of common usage patterns. See https://github.com/jonathandturner/rfcs/blob/b4ff67a2406651372c72fe7af856c3af62846e67/text/0000-api-doc-frontpage-styleguide.md for the kinds of things the crate root docs should discuss.

Failed to build under i686-pc-windows-gnu

AppVeyor build log: https://ci.appveyor.com/project/zonyitoo/context-rs/build/job/628poi9n7jr46ga2

Error log:

   Compiling memmap v0.1.0
     Running `rustc C:\Users\appveyor\.cargo\registry\src\github.com-121aea75f9ef2ce2\memmap-0.1.0\src\lib.rs --crate-name memmap --crate-type lib -g -C metadata=ea906f06964a25e2 -C extra-filename=-ea906f06964a25e2 --out-dir C:\projects\context-rs\target\debug\deps --emit=dep-info,link -L dependency=C:\projects\context-rs\target\debug\deps -L dependency=C:\projects\context-rs\target\debug\deps --extern bitflags=C:\projects\context-rs\target\debug\deps\libbitflags-d8aad5e08369cb92.rlib --extern winapi=C:\projects\context-rs\target\debug\deps\libwinapi-3afcd2170e6034e1.rlib --extern kernel32=C:\projects\context-rs\target\debug\deps\libkernel32-c606ce60307b6481.rlib --extern libc=C:\projects\context-rs\target\debug\deps\liblibc-165b5479aec0459c.rlib --cap-lints allow`
C:\Users\appveyor\.cargo\registry\src\github.com-121aea75f9ef2ce2\memmap-0.1.0\src\windows.rs:109:57: 109:67 error: mismatched types:
 expected `u32`,
    found `u64`
(expected u32,
    found u64) [E0308]
C:\Users\appveyor\.cargo\registry\src\github.com-121aea75f9ef2ce2\memmap-0.1.0\src\windows.rs:109                                                         len as u64) };
                                                                                                                                                          ^~~~~~~~~~
C:\Users\appveyor\.cargo\registry\src\github.com-121aea75f9ef2ce2\memmap-0.1.0\src\windows.rs:109:57: 109:67 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
Could not compile `memmap`. 

Separate virtual page allocation from memory maps backed by page file

On linux both of these are done using mmap but on Windows these are very different. Memory maps are done using CreateFileMappingW and can be done either from an actual file or the page file, while virtual page allocation is done using VirtualAlloc which is significantly more flexible. The only reason you'd have to use a memory map backed by the page file on Windows however is because it is an NT object technically and can thus be shared with other processes making it useful for IPC.

How to access char devices?

I a have a need to acce physical memory on arm device. I want to use memap to open /dev/mem but it fails with panic

thread 'main' panicked at 'failed to map the file: Error { repr: Custom(Custom { kind: InvalidInput, error: StringErro
r("memory map must have a non-zero length") }) }', /checkout/src/libcore/result.rs:906:4

It looks like /dev/mem has zero length for memmap. How can I handle that situation?

Overhaul API

• Remove MmapView and MmapViewSync types
  • While it is very common to need something like these types when working with memory maps, the exact semantics tend to differ enough that you can't actually use these types. In addition, they're simple enough that removing them won't have too much of a negative impact.
• Move the "unsafety location" from access to construction
  • Define two types - Mmap and MmapMut for readable and readable/writable mappings respectively. They implement Deref<Target= [u8]>/DerefMut as appropriate. In contexts where access to the mapped region could cause faults (e.g. file backed maps on Unix) make the constructor unsafe.
• Keep a couple of convenience constructors on Mmap and MmapMut, but move advanced configuration to an OpenOptions-style builder. The builder will provide a cross-platform API directly with OS specific extension traits.

Huge Pages

What is required to get support for huge pages? Is it just a matter of passing the correct flags to mmap with some checks around alignment and size etc?

Clarification

Thanks for your work on this library; very useful indeed! I just had a few things I was wondering about:

• Are MmapView/MmapViewSync simply for allowing different owners of different parts of the mmap? (I presume this doesn't cause any fragmentation of the underlying map.)
• In what sense is MmapViewSync more 'thread-safe' than MmapView? Accessing/modifying data seems to be equally un-thread-safe from both, so what's the difference?
• For the clone method, you state The underlying memory map is shared, and thus the caller must ensure that the memory underlying the view is not illegally aliased. – what sort of aliases are you referring to here?

Build error with rust 1.21

Works on 1.20, fails on 1.21 with the following error:

     Running `/usr/bin/rustc --crate-name cat examples/cat.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=e6f5be159863a10d -C extra-filename=-e6f5be159863a10d --out-dir /builddir/build/BUILD/memmap-0.5.2/target/release/examples -L dependency=/builddir/build/BUILD/memmap-0.5.2/target/release/deps --extern tempdir=/builddir/build/BUILD/memmap-0.5.2/target/release/deps/libtempdir-6b2c81b785a9b5c4.rlib --extern libc=/builddir/build/BUILD/memmap-0.5.2/target/release/deps/liblibc-32609fb12466c024.rlib --extern memmap=/builddir/build/BUILD/memmap-0.5.2/target/release/deps/libmemmap-f8b77f32bb5386b7.rlib -Copt-level=3 -Cdebuginfo=2 -Clink-arg=-Wl,-z,relro,-z,now`
error: variable does not need to be mutable
   --> src/lib.rs:880:17
    |
880 |             let mut jitmem = unsafe { map.as_mut_slice() };
    |                 ^^^^^^^^^^
    |
note: lint level defined here
   --> src/lib.rs:3:9
    |
3   | #![deny(warnings)]
    |         ^^^^^^^^
    = note: #[deny(unused_mut)] implied by #[deny(warnings)]

error: aborting due to previous error

error: Could not compile `memmap`.

Expose a low-level API in lib.rs

I'd love to use this library, but currently it's doing too much for my particular use case.
In particular, I'd like to:

• mmap any offset in a file to a precise location in memory (I know what page boundaries are, I'm ok with handling errors). Basically, any extra checking is going to cost a lot in some of my use cases.
• flush and unmap whenever I want (the order in which I do it are fundamental to my use case).

Add an `map_copy_read_only` API

It's useful to be able to map a file PROT_READ and MAP_PRIVATE on Linux. Later on you can make some pages writeable and write them, or use /proc/.../mem to write them. Currently memmap can do this but it takes two steps: a map_copy followed by make_read_only, during which there's a short window when the pages are writeable.

Document error conditions for fallible methods in a separate "Errors" section

Add examples to methods

See https://github.com/brson/rust-api-guidelines#c-example for details.

Note that this should be sequenced after #33.

MAP_POPULATE on linux?

Can be usefull for mapping files.

Expose page_size pub function as part of crate api

Currently I'm creating my pages using a non cross platform independent API set. I'm basically using a modified version of this crate where page_size is exposed for windows and linux. I'm unsure of how to do this same thing on some of the other platforms memmap-rs supports or if it's the same as the linux call.

Would you be interested in a pull request with these calls public and cross platform implemented?

File ownership

It seems to me that the intended semantic in unix::MmapInner::open would be file.into_raw_fd() instead of file.as_raw_fd(). I'm not sure how mmap behaves in practice if the file descriptor is pointing to a closed file (if it was maped when open), but it seems like good abstraction to move the ownership of the file when mapping it.

Note that this would imply the following changes to unix::MmapInner:

• adding a field keeping track of the file descriptor
• changing the signature of open(file: &File, ...) to open(file: File, ...) (would also need to propagate to the user facing signatures)
• closing the file in the Drop implementation, after unmaping the memory

what happens if we exceed isize::MAX on 32-bit platforms?

The ptr::offset function, which underlies safe slice indexing, says in its docs:

If any of the following conditions are violated, the result is Undefined Behavior...The computed offset, in bytes, cannot overflow an isize...memory acquired directly from allocators or memory mapped files may be too large to handle with this function.

Since the map function is unsafe, it's arguably fine for it to expose possible UB in this way. But I think most people reading the docs won't have any idea that this is a requirement. Maybe it would be better for Deref to panic rather than to return a slice that's "unsoundly large"? (Edit: Probably just return an error if we try to mmap something larger than isize::MAX?)

Triage the issue tracker

It would be helpful to label and prioritize issues that are blocking a 1.0 release of memmap. Here is what I am able to tell:

• #2—From a skip through the code, this looks like a backward compatible addition. Is this a blocker for 1.0? It has been 2 years. Do we still want something like this? If a different approach would be better, let's close the PR and file an issue to track it.
• #18—Looks like this has been resolved, possibly with a documentation clarification outstanding. Let's file a separate issue to track specifically the docs work.
• #21—How likely is this to happen? Is it a blocker for 1.0? Can it be backward compatible?
• #25—This needs to be documented better. I would consider this important for 1.0.
• #26—Is this a blocker for 1.0? Can it be backward compatible?
• #29—Do we still want an mprotect wrapper in memmap? Is it a blocker for 1.0?
• #30—How likely is this to happen? It won't work on Windows.
• #31—This seems like a high priority to address before stabilizing anything.
• #32—This needs work before 1.0 but is blocked by #33.
• #33—This may or may not be done. All the checkboxes are checked. If there is remaining work, it should be tracked in a separate issue.
• #34—This should have been closed.
• #37—This needs work before 1.0 but is blocked by #33.
• #46—Unclear what the remaining work is. Please explain in a comment in case somebody else is motivated to pick it up. This seems like a blocker for 1.0.
• #47—I agree that "quite a bit more documentation would be useful," even if that means linking heavily to system man pages. I feel this needs attention before 1.0.
• #48—This would be a breaking change so it needs to be settled before 1.0.
• #49—Looks like this has been resolved and no change is needed.

Support discontinuous memory maps

I'm working with a file format that scatters its data around on disk. It's composed of multiple logical data streams which are intended to be memory mapped. The segments are all page-aligned, and records within a stream constantly refer to other records based on their positions within the stream entirely oblivious to the physical layout.

Ultimately I want a single continuous read-only &[u8] given a list of offsets and sizes:

0x462C000:   0x1000 bytes
0x473D000: 0x8C4000 bytes
0x5003000: 0xFFE000 bytes
0x6003000: 0x29E000 bytes

In the case of POSIX mmap(), I think this would be done by using mmap(NULL, <total size>, MAP_ANON, PROT_NONE, 0, 0) to ask for a chunk of address space without actually committing resources to it, followed by repeated mmap(<address>, <size>, MAP_FIXED, PROT_READ, <fd>, <offset>) to map each file piece to the appropriate location within that space.

On Windows, I think this is VirtualAllocEx(MEM_RESERVE) to get the address space, then MapViewOfFileEx() to map each piece.

Is this crate still maintained?

It seems like this crate wasn't really (well?) maintained since October 2018.
Last commit was 3b047cc (18 Apr 2018) and last release was memmap v0.7.0 (20 Sep 2018).

Execute protection

Add execute protections to

memmap::Protection

to allow use in a JIT-style use case

Add #[doc(html_root_url = "...")] annotation to crate root

Empty Files not working

Hi, there seems to be an issue when, a Mmap is created and on an empty file.
I get an "The volume for a file has been externally altered so that the opened file is no longer valid." error under windows.

You can add this to the tests to reproduce.
I also noticed the offset test is not working on Windows.

use std::fs::File;
#[test]
fn map_empty_file_copy() {
    let tempdir = tempdir::TempDir::new("mmap").unwrap();
    let path = tempdir.path().join("mmap");

    let file = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open(&path)
        .unwrap();

    let get_mmap = |file:&File| -> Mmap {
        unsafe { Mmap::map(&file).unwrap() }
    };

    let mmap = get_mmap(&file);
}

Please update to fs2 0.4

fs2 has a new version 0.4 available; please consider updating Cargo.toml to allow it. This shouldn't affect the API of memmap in any way, but it would allow users of memmap that need fs2 0.4 to avoid building and including two separate versions of fs2.

Tracking issue for crate evaluation

• Expand crate-level documentation
• Add examples to methods
• Overhaul API
• Add CI Badges to Cargo.toml
• Add #[doc(html_root_url = "...")] annotation to crate root
• Document error conditions for fallible methods in a separate "Errors" section

Incorrect behavior if file length can't fit in usize

In this line (found twice in lib.rs):

let len = try!(file.metadata()).len() as usize;

len() returns a u64; when encountering a 4GB+ file on a 32-bit platform, this will silently chop off the high bits of the length and map some prefix of the file whose length corresponds to the low bits. It would be better to fail in this case.

data-race safety

The API exposed by Mmap is not data-race free. For example, if two processes or threads mmap the same file and proceed to read and write, the assumptions Rust makes about &mut [u8] are broken, and the behavior is undefined. It is probably impossible to statically verify that mmap usage is data-race free, however it is against all Rust conventions to expose an API not marked unsafe which can trigger undefined behavior.

Open questions:

• What is the best way to limit the potential unsafety?
• Are data races confined to just shared mmaps, or do anonymous maps expose data races?
• What API changes should be made?

Should anonymous mappings be MAP_SHARED by default?

Currently they are on unix, at least based on a reading of the source. This seems like a kinda surprising footgun, especially as memmap is potentially an indirect dependency in a bunch of use cases. Creating a shared memory segment between parent and child should probably be an explicit action in MmapOptions?

Allow anonymous RWX mappings

Currently it doesn't seem to be possible to create an anonymous RWX mapping. Could we get something like make_exec_mut?

Error when mapping large files on Windows

When I try to map files 2.1 GB or larger on Windows 10 I get errors.

Code looks like this:

extern crate memmap;

use memmap::MmapOptions;
use std::fs::File;
use std::io::{self, Error, Write};

fn main() -> Result<(), Error> {

    let path = "<large file>";

    let file = File::open(path)?;
    let mmap = unsafe { MmapOptions::new().map(&file)? };

    io::stdout().write_all(&mmap[..10])?;

    Ok(())
}

• With a 2.1 GB file I get Error: Os { code: 8, kind: Other, message: "Not enough memory resources are available to process this command." }.
• With a 16.6 GB file I get Error: Custom { kind: InvalidData, error: "memory map length overflows usize" }
• Works file with a 1.1 GB file.

My machine has 32 GB memory and opening the files as memory maps in e.g. Python works just fine. I'm new to Rust, so perhaps I'm missing something obvious...

Possible overflow while mapping large files on 32 bit systems

memmap-rs/src/lib.rs

Line 129 in 9df1548

Ok(len as usize)

When mapping a file > 4GiB on a 32bit system, this line will produce an integer overflow resulting in a more-or-less "random" mapping length. The correct behaviour here would be to return an error if the requested mapping is larger than the address space.

Add MAP_FIXED mode

Add a construction option for creating MAP_FIXED memory maps at a known memory location. CC #20 .

Implement Read + Seek for Views?

Seems to me that View could implement these pretty effortlessly, though it might collide somewhat with split_at. I assume Seek would ignore splits and work in terms of underlying map, though.