Memory leak in http server

[rust-lang]

Memory leak in http server

[kamalmarhubi]

I don't think this is in the http server. A simple qotd server is growing about 500 bytes per request. Source: https://github.com/kamalmarhubi/futures-rs/blob/qotd/mio/examples/qotd.rs

[kamalmarhubi]

After @jvns showed me ltrace last night, we tracked down some leaking allocations and found that in fact the very first accept(listener) allocation is never freed.

In both cases, the heap growth is from holding on to the initial accept(listener) future, which is await'ed. This future never completes: instead a chain of futures is created via join, which grows with each request received. Using join ensures we listen for the next request, but at the cost of making the current request's future not complete until the next request has. And so on.

It's easiest to see in the qotd example:

fn main() {
    // ...
    l.await(accept(listener)).unwrap();
}

fn accept(listener: TcpListener) -> Box<IoFuture<()>> {
    listener.accept().and_then(move |(stream, _addr)| {
        send_quote(stream)
            .join(accept(listener))  // this is problematic!!
            .map(|_| ())
            .boxed()
    }).boxed()
}

fn send_quote(stream: TcpStream) -> Box<IoFuture<()>> {
    let quote = b"If I knew any quotes, I'd've put one here.".to_vec();
    stream.write(0, quote).map(|_| ()).map_err(From::from).boxed()
}

I think the pub fn accept(&self) -> Box<IoFuture<(TcpStream, SocketAddr)>> API is what is "wrong" here: wrapping this up in a future is conceptually nice but has issues. I'm experimenting with a replacement listening API like this:

impl Loop {
    pub fn start_server<F>(&mut self, addr: &SocketAddr, cb: F)
                           -> io::Result<()>
        where F: Fn(TcpStream, SocketAddr) -> Box<IoFuture<()>> + 'static
}

This works for the qotd server, keeping memory usage steady over at least sequential requests. (This wasn't true previously!) For the HTTP server, sequential requests have memory usage steady, but concurrent requests just don't work right now. I think I messed something up in socket registration.

[kamalmarhubi]

All this is to say: the leak is not in the core futures lib!

[alexcrichton]

Nice investigation! That definitely makes sense to me. Definitely a not-so-good design :)

I'll close this for now as it shouldn't affect the core futures lib and just the toy stuff I've been doing should probably change...