Using compression in spdy server push

[expressjs]

Using compression in spdy server push

[dougwilson]

Hmm, I don't think this module works with SPDY push, though I also think no Express middleware does, either (middleware that alters the response). I'm not too familiar with SPDY push, though.

Would you be willing to post an example or even work on a PR to add this :)?

[hdf]

I could try, but I might need some pointers. Like which parts of the code to focus on. Should also figure out first how exactly we want this to work.
My current idea is something like this:

var compress = require('compression');
var file_name = 'file.js';
var file = fs.readFileSync(file);
var compressed = compress(file_name, file);
...

  var stream = res.push('/' + file_name, compressed.headers, pushHandler);
  compressed.data.pipe(stream);

What do you think?

[dougwilson]

That looks like a different module; this module is only an Express middleware.

[hdf]

But I would like to reuse a lot of what compression does, like determine, if the file should be compressed at all, and what headers to set based on that. Maybe if it's like this?:

var compresser = require('compression').compresser;
var compressed = compresser(file_name, file);

[dougwilson]

Still no; if you extract that stuff as another generic module, we can simply use your lower-level module. The only interface this module will have an an Express middleware interface.

[hdf]

So you want me to make something, that compression itself will be based on? Wow, that's a big request, I'm still a newbie to this, not feeling very confident.

It's a bit weird, that currently we can't conveniently use compression in a push stream. Makes the whole thing kind of pointless (server pushes). Or so it feels.

[dougwilson]

Right, though I did intend to do this eventually, if you don't mind waiting, and are hesitant to do it :) When I get to a computer, I do believe there may be a way to at least temporarily hack an Express middleware to manipulate a stream, though would need to know how to use SPDY push if you can provide an example.

I think you would find that implementing that extra export as a non-hack would mean rewriting the internals of this module anyway :)

[dougwilson]

It's a bit weird, that currently we can't conveniently use compression in a push stream. Makes the whole thing kind of pointless (server pushes). Or so it feels.

I agree, but have no experience using them, so have no idea how to go about changing this or even how to make a server that does push and understand how to make a request to said server to see the result of the push, if you can teach me?

[hdf]

Currently for client side testing I'm just using the latest Firefox and Chrome, and their DevTools (ctrl+shift+i). (Can disable caching while DevTools are out.)
My current server code for push is probably pretty embarrassing, but here it is commented out:
https://github.com/hdf/clusters/blob/master/server/index.js#L111
The relevant bit is this:

function pushIt(file) {
  var s = res.push('/static/' + file, mime(file), pushHandler);
  var bufferStream = new stream.Transform();
  bufferStream.push(push_cache[file]);
  bufferStream.end();
  bufferStream.pipe(s);
}

push_cache[file] is a Buffer object.

Another idea: Is it possible, to somehow "virtually" trigger compression, and feed what it gives to that push function and stream?

[dougwilson]

Another idea: Is it possible, to somehow "virtually" trigger compression , and feed what it gives to that push function and stream?

That sounds like the "hack" I was thinking about, yes.

[hdf]

OK, I have solved this for my self, and it is now clear, that this is not an express compression issue. It is best, if files that should be pushed are cached in a compressed state. And that is really contrary to how this module works.