softprops/hyperlocal

🔌 ✨rustlang hyper bindings for local unix domain sockets

u

🔌 ✨

hyperlocal

Hyper client and server bindings for Unix domain sockets

Hyper is a rock solid Rust HTTP client and server toolkit. Unix domain sockets provide a mechanism for host-local interprocess communication. hyperlocal builds on and complements Hyper's interfaces for building Unix domain socket HTTP clients and servers.

This is useful for exposing simple HTTP interfaces for your Unix daemons in cases where you want to limit access to the current host, in which case, opening and exposing tcp ports is not needed. Examples of Unix daemons that provide this kind of host local interface include Docker, a process container manager.

Installation

Add the following to your Cargo.toml file

[dependencies]
hyperlocal = "0.9"

Usage

Servers

A typical server can be built by creating a tokio::net::UnixListener and accepting connections in a loop using hyper::service::service_fn to create a request/response processing function, and connecting the UnixStream to it using hyper::server::conn::http1::Builder::new().serve_connection().

hyperlocal provides an extension trait UnixListenerExt with an implementation of this.

An example is at examples/server.rs, runnable via cargo run --example server

To test that your server is working you can use an out-of-the-box tool like curl

$ curl --unix-socket /tmp/hyperlocal.sock localhost

It's a Unix system. I know this.

Clients

hyperlocal also provides bindings for writing unix domain socket based HTTP clients the Client interface from the hyper-utils crate.

An example is at examples/client.rs, runnable via cargo run --example client

Hyper's client interface makes it easy to send typical HTTP methods like GET, POST, DELETE with factory methods, get, post, delete, etc. These require an argument that can be transformed into a hyper::Uri.

Since Unix domain sockets aren't represented with hostnames that resolve to ip addresses coupled with network ports, your standard over the counter URL string won't do. Instead, use a hyperlocal::Uri, which represents both file path to the domain socket and the resource URI path and query string.

---

Doug Tangren (softprops) 2015-2020

hyperlocal's Issues

Update for hyper 0.12?

There's a new hyper/tokio out.

They've changed everything, and there's no upgrade documentation.

Uh, strike that. I mean, it'd be fun to add support for it!

Cargo release 0.9.0

Happy to see this crate integrate with the latest Hyper release in #65 . This is working well with my testing. Is there any remaining work to publish to crates.io ?

Would be keen to see a 0.9.0 release. Grateful for any time spent on this.

@softprops

Work greetly with axum, but not required in hyper 1.0

I do not know why I spent all this morning for just open a unix socket, anyway, the full example ported from TcpListener to UnixListener:

use bytes::Bytes;
use http_body_util::Full;

use hyper::server::conn::http1;
use hyper::service::service_fn;
use hyper::{Request, Response};
use tokio::net::{TcpListener,UnixListener};
use hyper_util::rt::TokioIo;

use std::convert::Infallible;


async fn hello(_: Request<hyper::body::Incoming>) -> Result<Response<Full<Bytes>>, Infallible> {
    Ok(Response::new(Full::new(Bytes::from("Hello, World!"))))
}

#[tokio::main]
async fn main() {
    let socketpath = "/tmp/notexistent.sock";
    let path = std::path::Path::new(socketpath);

    if path.exists() {
        tokio::fs::remove_file(path).await.expect("Could not remove old socket!");
    }

    let listener = UnixListener::bind(path).unwrap();

    //let listener = TcpListener::bind(addr).await.unwrap();
    loop {
        let (stream, _) = listener.accept().await.unwrap();
        let io = TokioIo::new(stream);
        
        tokio::task::spawn(async move {
            if let Err(err) = // http1::Builder::new()
            http1::Builder::new().serve_connection(
                io,
                service_fn(hello),
            )
            .await
            {
                println!("Failed to serve connection: {:?}", err);
            }
        });
    }
}

try with:

curl --no-buffer -XGET --unix-socket /tmp/notexistent.sock http://domain/saysomething

I refer to the example in https://hyper.rs/guides/1/server/hello-world/

Does not compile if client-only

In my crate I have the following in my Cargo.toml

hyperlocal = { version = "0.7", features = [ "client" ], default_features = false }

I get the following error:

   Checking hyperlocal v0.7.0
error[E0433]: failed to resolve: unresolved import
  --> /home/nate/.cargo/registry/src/github.com-1ecc6299db9ec823/hyperlocal-0.7.0/src/lib.rs:41:16
   |
41 | pub use crate::server::conn::SocketIncoming;
   |                ^^^^^^
   |                |
   |                unresolved import
   |                help: a similar path exists: `hyper::server`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: could not compile `hyperlocal`.

This would appear to be because https://github.com/softprops/hyperlocal/blob/master/src/lib.rs#L41 is not preceded with #[cfg(feature = "server")]

is this still maintained ?

Vulnerability GHSA-q6cp-qfwq-4gcv in hyperlocal's dependency - h2

Summary

The dependency h2(0.3.24) has vulnerability issue, check details from GHSA-q6cp-qfwq-4gcv

Details

The dependency h2(0.3.24) has vulnerability issue, check details from GHSA-q6cp-qfwq-4gcv
Fixed in 0.4.4

The hyper has new version fixed this. Need to update.

Dependency Tree

├─┬ hyperlocal 0.8.0 - Cargo
│ └─┬ hyper 0.14.28 - Cargo
│ └── h2 0.3.24 - Cargo

upgrade to hyper 0.9

this involves upgrading url to 1.0 which had some design changes. They should be straightforward to work around given some feedback mentioned here

cannot use with tokio::run()

There's no way to get Futures from incoming connections. Plain old hyper uses this kind of code:

let s = Server::bind(&addr.parse().unwrap())
  .serve(new_service)
  .map_err(|e| eprintln!("server error: {}", e));
hyper::rt::run(s);

But hyperlocal doesn't allow it. Even if you do into_future() (after mapping the outputs to ()), then it doesn't work because it never actually processes requests.

This feature is necessary because hyperlocal's event loop doesn't work at all with threadpooling and the tokio_threadpool::blocking function.

Client access websocket without proxy

I am using hyperlocal and hyper-tungstenite to have a websocket over UDS. For a client to access such websocket I currently proxy the websocket to localhost with something like socat TCP-LISTEN:12345 UNIX-CONNECT:/tmp/hyperlocal.sock
Then I regularly connect to ws:///localhost:12345

However, I would like to avoid proxying the websocket and access it directly via /tmp/hyperlocal.sock.
I still need a websocket not a http GET/POST request.

Any hint?

RFC: Release 0.9

There have been a couple of small quality-of-life improvements since the previous release. What I am personally interested in is the compilation time reduction in the main branch over 0.8. Some of the documentation would need updating to match the fact that features are no longer enabled by default.

Use either rustfmt.toml or .rustfmt.toml but not both

It seem that in this repository there is both a rustfmt.toml and a .rustfmt.toml file which is confusing as only one will be used. See this report

UnixListener::from_std causes the server to hang

This code used to work fine in hyperlocal 0.7 and the old tokio. In hyperlocal 0.8 and tokio 1.x the service_fn callback never sees any requests coming, and the client is stuck waiting for a response forever.

In 0.8 the server works fine only with a tokio UnixListener created directly from a path. It hangs if the listener has been converted from its std version.

#[tokio::test]
async fn hyperserve() {
    use hyper::{Server, Client, Body, Response, service::{make_service_fn, service_fn}};
    use hyperlocal::{UnixClientExt, Uri};

    let socket_path = "/tmp/test-hyperlocal.sock";
    let _ = std::fs::remove_file(socket_path);

    let make_service = make_service_fn(|_| async {
        Ok::<_, hyper::Error>(service_fn(|_req| async {
            Ok::<_, hyper::Error>(Response::new(Body::from("It works!")))
        }))
    });

	// Tokio listener created directly from path works fine:
    // let listener = tokio::net::UnixListener::bind(socket_path).unwrap();

	// Tokio listener created via std listener never responds:
    let listener = tokio::net::UnixListener::from_std(
        std::os::unix::net::UnixListener::bind(&socket_path).unwrap()
    ).unwrap();
    let builder = Server::builder(SocketIncoming::from_listener(listener));

    tokio::spawn(builder.serve(make_service));

    let client = Client::unix();
    let uri: hyper::Uri = Uri::new(socket_path, "/").into();
    let res = client.get(uri).await.unwrap();
    assert_eq!("It works!", hyper::body::to_bytes(res.into_body()).await.unwrap());
}

http paths errors

I am using it to query podman unix socket and it requires paths such as http://d/v3.0.0/libpod/pods/json, which works fine in curl.
But of course such a path is invalid for hyper:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidUri(InvalidAuthority)'

Body is dropped

Hi,
I am trying to use this hyperlocal to send http api to my http server in below method:

    pub async fn put(&self, path: &str, data: Option<String>) -> Result<()> {
        let client = Client::unix();
        let uri = Uri::new(&self.sock_path, path);
        let (body, _) = if let Some(d) = data {
            let l = d.len();
            (d.into(), l)
        } else {
            (Body::empty(), 0)
        };

        let req = Request::builder()
            .method(Method::PUT)
            .uri(uri)
            .body(body)
            .unwrap();

        client.request(req).await.unwrap();

But my http user gets no body from the request.
Do we have examples on how to PUT/POST http request with body? Or did I do something wrong in above code snippet?

Thoughts about making UnixStream public

👋 I'm looking at using this logic as part of my own HTTP Client, and I was hoping to get access to the UnixStream type: https://github.com/softprops/hyperlocal/blob/main/src/client.rs#L19.

The main reason i'm doing this all is to honestly have hyper tuned error reporting. Certain IO Error codes mean certain things depending on the stream type, and by wrapping everything in my own custom enums I can make decisions/keep state about the streams, connectors, etc. In any part of my codebase.

I realize most people get fine without it though, and I imagine it was made private for a reason, so I just wanted to see if there was any chance we could start exporting that type.

Socket permissions.

Hi!

I have a scenario in which Apache is running as a reverse proxy to a hyper server through a UDS. Things seem to work well enough but there is an issue with the permissions of the socket. I need it to be writable by both user and group as I don't want to run both apache and the hyper server under the same user.

I tried to simply open the socket as a file and use "set_permissions". This doesn't work however as the socket file isn't created until server.run() is executed and then it's too late for me to do it in the program.

I'm sure it's doable in some way but I can't seem to figure it out. Preferably the socket should be created early enough that I could launch the program as root, create the socket and set whatever permission I need, drop any permissions and finish of by starting to listen to the socket.

Any ideas?

README.md example is out of date

It asks you to use hyperlocal::server::Server::new, but that's been removed: https://docs.rs/hyperlocal/0.6.0/hyperlocal/server/struct.Server.html

Server example doesn't use hyperlocal?

This may just be a misunderstanding on my part, but hyperlocal advertises itself using (emphasis added):

Hyper is a rock solid Rust HTTP client and server toolkit. [...] hyperlocal builds on and complements Hyper's interfaces for building Unix domain socket HTTP clients and servers.

Later in the README it gives a server example at examples/server.rs, but this example doesn't even seem to use hyperlocal. So I don't understand why hyperlocal advertises itself as being a server toolkit, while the server example doesn't use hyperlocal. This leaves me (as someone seeing the library for the first time) confused about what this project actually is or what it does.

get ready for hyper's 0.11 changes

some discussion is being had over here hyperium/hyper#126

Need to be able to chmod the socket

After I call Http::bind(), I need to be able to chmod the file descriptor appropriately.

In theory, something like this:

let s = hyperlocal::server::Http::new()
  .bind(&address["unix:".len()..], new_service)?;
libc::fchmod(s.listener.as_raw_fd(), 0o777);

But this isn't possible because s.listener is not pub.

Connection socket does not shut down cleanly

Hyper logs debug errors for every HTTP request made to a server backed by a Unix Socket.

Example output

2023-02-03T03:44:08.141436Z DEBUG hyper::proto::h1::conn: error shutting down IO: Socket is not connected (os error 57)
2023-02-03T03:44:08.141492Z DEBUG hyper::server::server::new_svc: connection error: error shutting down connection: Socket is not connected (os error 57)

Reproduction

Code:

use hyper::{service::{make_service_fn, service_fn}, Request, Body, Response};
use hyperlocal::UnixServerExt;
use std::{path::PathBuf, convert::Infallible};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[cfg(unix)]
#[tokio::main]
async fn main() {
    tracing_subscriber::registry()
        .with(
            tracing_subscriber::EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "debug".into()),
        )
        .with(tracing_subscriber::fmt::layer())
        .init();

    let path = PathBuf::from("/tmp/helloworld");

    let _ = tokio::fs::remove_file(&path).await;
    tokio::fs::create_dir_all(path.parent().unwrap())
        .await
        .unwrap();

    let make_service = make_service_fn(|_conn| async {
        Ok::<_, Infallible>(service_fn(handle))
    });

    hyper::Server::bind_unix(path).unwrap().serve(make_service)
        .await
        .unwrap();
}

async fn handle(_req: Request<Body>) -> Result<Response<Body>, Infallible> {
    Ok(Response::new(Body::from("Hello World")))
}

Cargo deps:

hyper = { version = "0.14", features = ["full"] }
tokio = { version = "1.25", features = ["full"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
hyperlocal = "0.8.0"

To reproduce, run the code and connect to the socket, with e.g.

curl --unix-socket /tmp/helloworld something

Additional thoughts

It's just mildly annoying, as far as I can tell, it doesn't cause any issues. I'm fairly new to async stuff in Rust, but I tried to dig into it. I'm not even sure if this is a hyperlocal, hyper or tokio problem.

The error seems to be produced by poll_shutdown: https://github.com/hyperium/hyper/blob/499fe1f949895218c4fd2305a0eddaf24f1dd0a9/src/proto/h1/conn.rs#L720. Tracing that through, it just calls shutdown on the UnixStream. I've think it's unlikely to be a hyper problem, because that is transport agnostic and doesn't have any issues when listening on a port.

Is there something that hyperlocal could do to handle the connection socket shutdown (better)?

Backport hyperlocal windows

https://github.com/Azure/hyperlocal-windows

Impl drop for server which deletes the associated socket file

It's really not intuitive to require users to remove this file themselves. Also not intuitive ux when stopling and starting servers

SocketIncoming is not exported

bind_unix returns Builder<SocketIncoming>, but SocketIncoming isn't exported from the crate, so I can't name it to return it from my functions, put it in a struct, etc.

low level example server is broken

I'd like to be able to use the lower level api for servers as is described in this source code comment: https://github.com/softprops/hyperlocal/blob/master/src/server/mod.rs#L163-L207

Below i added a main function for that example code.... and.... it doesn't work. It compiles. It runs and exits immediately.

extern crate hyper;
extern crate hyperlocal;

use std::os::unix::net::UnixListener;
use hyper::{Response, rt::{Future, Stream}, service::service_fn};
use hyperlocal::server::{Http, Incoming};


fn main() {
    if let Err(err) =  std::fs::remove_file("hyperlocal_test_echo_server_2.sock") {
        if err.kind() != std::io::ErrorKind::NotFound {
            panic!("{}", err);
        }
    }

    let listener = UnixListener::bind("hyperlocal_test_echo_server_2.sock").unwrap();
    let incoming = Incoming::from_std(listener, &Default::default()).unwrap();
    let serve = Http::new().serve_incoming(
        incoming,
        move || service_fn(
            |req| Ok::<_, hyper::Error>(Response::new(req.into_body()))
        )
    );

    let server = serve.for_each(|connecting| {
        connecting
            .then(|connection| {
                let connection = connection.unwrap();
                Ok::<_, hyper::Error>(connection)
            })
            .flatten()
            .map_err(|err| {
                std::io::Error::new(
                    std::io::ErrorKind::Other,
                    format!("failed to serve connection: {}", err),
                )
            })
    });
}

Incorrect Host header is set

Requests made via hyperlocal have a Host header with a hostname of the "fake" socket URL, rather than the original hostname of the URL being wrapped, e.g.

Host: 2f7661722f666f6c646572732f31332f633839683062316e376e64326c7432733433675f64765f3830303030676e2f542f2e746d706265417941472f746573742e736f636b:0

instead of

Host: example.com

depend on rust-hex not rustc-serialize

the last release added a dep on rustc-serialize just for hex. it turns out, there's a crate for that

https://github.com/KokaKiwi/rust-hex/blob/master/src/lib.rs

let's use this instead