schveiguy/io

Core I/O functionality

Documentation std.io

IOs are thin, OS-independent abstractions over I/O devices.

size_t write(const scope ubyte[] buffer);
size_t read(scope ubyte[] buffer);

IOs support scatter/gather read/write.

size_t write(const scope ubyte[][] buffers...);
size_t read(scope ubyte[][] buffers...);

IOs are @safe and @nogc.

void read() @safe @nogc
{
    auto f = File(chainPath("tmp", "file.txt"));
    ubyte[128] buf;
    f.read(buf[]);
    // ...
}

IOs use exceptions for error handling.

try
    File("");
catch (IOException e)
{}

IOs use unique ownership and are moveable but not copyable (Use refCounted for shared ownership).

io2 = io.move;
assert(io2.isOpen);
assert(!io.isOpen);

auto rc = refCounted(io2.move);
auto rc2 = rc;
assert(rc.isOpen);
assert(rc2.isOpen);

IOs can be converted to polymorphic interfaces if necessary.

Input input = ioObject(io.move);

io's Issues

Add Unix Socket support

Unix socket addresses should be at least supported for Posix drivers.

Take ownership unused?

I was browsing the source code and saw
https://github.com/MartinNowak/io/blob/ccc83f734a0801a3da9ab731b8e49bd7b5f00241/src/std/io/file.d#L165-L176
where the takeOwnership parameter is unused and closeOnDestroy is always set to false, is that correct behavior? It does not match with the current documentation

"IOs are `@nogc`" ... "IOs use Exceptions for error handling"

These two goals conflict. I think the easier option is to drop the @nogc requirement. Especially since D classes are built around the GC, and you are using classes for the drivers.

Provide a EOF function

I know it's trivial to just check returned value from read or write but it would be nicer syntactically and mentally.

Windows scattered sync read/write incorrectly implemented

The caller is expecting read to fill in the buffers sequentially. For a normal file, this may work properly, but for something other than a straight file, it's possible to read less than the requested buffer, but still have more data to read. This will result in garbage data being treated as real data.

Either the read should terminate when it does not read a full buffer (probably the right answer), or the reads should continue on the same buffer until it is full.

Also, a read of 0 bytes should terminate the loop.

dmd 2.089.0 reports error

There is an error with cas in dmd 2.089.0. The cas call should work as is in std.io, but fails. There are workarounds, but I'm expecting the bug to be fixed (https://issues.dlang.org/show_bug.cgi?id=20359). Once it's fixed, I'll fix io.

Sockets can't be moved

With dmd 2.083.0, I can't move a Socket using std.algorithm.move. I think this is a compiler bug, but I'm not sure why this fails.

I can call Socket.move no problem, but I can't call std.algorithm.move(Socket).

I can easily duplicate the problem in a small test case with:

import std.algorithm : move;

struct S
{
    int x;
    S foo() return nothrow @nogc pure { return S(x); }
    this(this) @disable;
}

void main()
{
    S s(2);
    auto s2 = s.foo; // ok
    auto s3 = move(s); // errors
}

Here are the errors (I removed some trivial path info):

std/algorithm/mutation.d(1232): Error: struct `testmove.S` is not copyable because it is annotated with @disable
std/algorithm/mutation.d(1225): Error: template instance `std.algorithm.mutation.moveImpl!(S)` error instantiating
std/algorithm/mutation.d(1183):        instantiated from here: trustedMoveImpl!(S)

The really odd thing is that the a member function MUST BE THERE, and it MUST have return attribute on it.

I will file an equivalent bug in bugzilla with this test case.

Rename repository?

the 'io' repository is used by https://github.com/jasonwhite/io on the dub registry. Even if I dub add-local this git directory, then I can't depend on it, dub still tries to download the other library. I believe it confuses the registration as pointing at the same repository, so if I don't specify the dependence on ~master (which AFAIK, is deprecated functionality), then it's going to go find the latest release online.

Not sure if there's a good path forward, because 'io' is a good name for this repository, and the other library is not being maintained.

Maybe 'basicio'?

Appveyor should be updated to the latest version of DMD

v0.2.5 worked around the CAS issue. AppVeyor is hard-coded to use 2.088.1, we should switch to latest.

See #28 (comment)

Add me to AppVeyor controls

@MartinNowak can you set it up so I have control over the AppVeyor instance? I can't restart tests without force-pushing, and it seems to have a lot of environmental failures.

FIXME: Using the imported std.internal.cstring.tempCStringW produces

https://github.com/MartinNowak/io/blob/14819fa88016d3f7dcdbe7172c3447e09f5f46a9/src/std/io/internal/string.d#L189

FYI: dlang/phobos#5932 (this will be fixed in 2.078)

String.clone is unsafe

Current implementation of String.clone does not duplicate the string, it simply sets a new reference. If the original string is freed or reallocated, the new clone is now dangling.

Drive the accepted socket from a separate thread

Is there a way to do that, possibly with std.concurrency?

I've tried using a standard ala C way to setup a listening socket, transferring the socket FD via concurrency.send and instantiating a new std.io.net.socket.Socket(fd) and everything seems to work.

I can't concurrency.send the std.io.net.socket.Socket` as the compiler complains about Aliases to mutable thread-local data

Setting SO_NOSIGPIPE on Socket for BSD derivatives

Under macOS, for example, If the socket for whatever reason is closed, you end up in the SIGPIPE handler.

Setting the SO_NOSIGPIPE option will instruct the OS to not generate SIGPIPE, but instead return EPIPE, which can be handled gracefully along with other IO errors.

Possibility of using file handle without creating ownership?

In iopipe I have a bool when creating a stream object:

this(int fd, bool closeOnDestroy = true)

Which tells the object whether or not to close the fd when the object is destroyed.

For things like stdin and stdout, this is almost necessary, as you don't want to close them just because you are done using them, other things may want to use them that aren't D.

But I see no way to duplicate this functionality if iopipe switched to using this library. Thoughts?

Add pipe handling.

Pipes are standard on all Operating systems. Though they differ in how created, generally there are mechanisms to create named pipes and unnamed pipes. We need this functionality for parent/child process communication if this is to replace the phobos i/o system.

prinf with string interpolation

I would like to have printf function with support of string interpolation (I am not sure that it's named so). All languages have it. Something like:
string name = "Mike";
int age = 20;
printf("Hello, {0}, your age is {1}", name, age);

Use with a growing buffer

How can I use this package with a growing buffer?

I tried a dynamic array, but it doesn't work

    ubyte[] buf;
    auto file = File(filePath, Mode.readWrite);
    file.read(buf);

    import std: writeln;
    writeln(buf); // buf is empty

can't build on windows with dmd 2.086.0

dub build --arch=x86_64
Performing "debug" build using C:\D\dmd2\windows\bin\dmd.exe for x86_64.
io 0.2.2: building configuration "library"...
src\std\io\file.d(357,20): Error: std.io.file.File.__ctor called with argument types (void*) matches both:
src\std\io\file.d(169,9):     std.io.file.File.this(void* handle)
and:
src\std\io\file.d(365,5):     std.io.file.File.this(return scope void* f)
C:\D\dmd2\windows\bin\dmd.exe failed with exit code 1.

Build failure, dmd 2.088: cast from int to void* not allowed in safe code

I have started seeing io package (v0.2.2) build failures when building jondegenhardt/dcat-perf. Builds are failing with an error in driver/package.d. The error occurs starting with DMD 2.088 / LDC 1.18.0. I suspect the build failure is a result of DMD PR #10056. The build error is:

$ dub build --compiler=ldc2
Performing "debug" build using ldc2 for x86_64.
io 0.2.2: building configuration "library"...
../../../../.dub/packages/io-0.2.2/io/src/std/io/driver/package.d(51,25): Error: cast from int to void* not allowed in safe code
../../../../.dub/packages/io-0.2.2/io/src/std/io/driver/package.d(95,27): Error: cast from int to void* not allowed in safe code

There are additional errors, but I suspect that they are a consequence of the initial errors. Compilation with DMD 2.087 / LDC 1.17.0 is fine.

The full build error log can be seen here: https://travis-ci.com/jondegenhardt/dcat-perf/builds/134325392. Look at the log of any of the failed builds.

Add to project tester for dmd

See here: #27 (comment)

Provide a way to open standard handles

Opening the standard handles on Windows and Posix is drastically different with low-level i/o. For D and C, you just use stdout/stdin/stderr, which are FILE * (or phobos File objects).

In io, you need to wrap the low-level handles/descriptors as File objects. We should provide a way to open a standard handle (maybe a D getStdHandle that forwards to the windows version, or returns 0, 1, 2 on Posix) to make cross-platform usage of the standard handles convenient.

We also may want to provide a way to use standard handles with a global, to prevent accidental closure of the handles, since IOs close themselves compulsively.

Github pages

The github pages deployment through travis is broken.
I think my message was not seen here

I think too page deployment is broken from a long time as travis documentation tell to set the secret variable from the settings page of io repository and to use this code:

deploy:
  provider: pages
  skip_cleanup: true
  github_token: $GITHUB_TOKEN  # Set in the settings page of your repository, as a secure variable
  keep_history: true
  on:
    branch: master

To meet the travis documentation we have to

1. set the secrete variable
2. remove before_deploy section
3. update the deploy section accordingly to the documentation

Need a way to close a socket read or write part individually

Sockets can be full duplex, we need a way to close individually the read and write end of the socket, as some protocols won't work unless you close your writing end first.

Investigate the viability of using `ReadFileScatter` and `WriteFileGather` for Windows scatter read

Requirement is that all the buffers are on a page boundary. WSASend and WSARecv do not have these requirements, so we can probably use those easier on sockets. See more info here: https://en.wikipedia.org/wiki/Vectored_I/O

Simpler DNS interface needed

The current DNS lookup interface is clunky to say the least. You need to pass in a callback that is immediately called. Much better to return a struct.

I'd recommend one of 2 things:

1. Keep the clunky interface, but add a wrapper that provides a resulting GC array of lookup addresses.
2. Instead of a callback, return a range that frees whatever resources it needs to when it's destroyed.

Doc publish authentication doesn't work

@MartinNowak this is for you. I don't know how this works internally, but all the changes I have merged so far have resulted in a failure to upload the docs. It says that it builds, but the authentication to push to the github pages fails.

`@safe` setter for driver

Track number of open files/sockets in driver to allow @safe setting/swapping of the IO Driver.

Add async driver

Goes without saying, we should have at least one of these. Possibly one that vibe.d can use, and I'd also like to see one based on mecca.

IPv4Addr constructor

The address will end up in network order only on little endian machine.