juliangruber/async-stream

The async stream spec

Hi, I'm Julian 👋

I'm a software engineer with a focus on P2P, Node.js, React, Blockchain and Electron.

Currently I'm creating Filecoin Station (desktop app) and Meridian (decentralized impact evaluator) at Protocol Labs.

async-stream's Issues

readme clarification

Could totally be misunderstanding but what you mean by this (specifically the bolded part)?

A generator stream is simply a generator function. If you're familiar with other stream semantics: There's only readable streams, no need for writable and transform.

add co-queue example

for working with pushing sources.

function wrap(emitter, event){
  var queue = new Queue;
  emitter.on(event, queue.push);
  return function*(end){
    if (!end) return yield queue.next;
  };
}

var read = wrap(emitter, 'data');
while (true) console.log(yield read());

add child stream error handling

add pull-cat example

suggested by @dominictarr, https://github.com/dominictarr/pull-cat

function cat(streams){
  var idx = 0;
  return function*(end){
    if (end) return streams[idx](end);
    var data;
    while (!data && streams[idx]) {
      try {
        // read
        data = yield streams[idx]();
      } catch (err) {
        // clean up
        streams.slice(idx).forEach(function(stream){
          try { yield stream(true) } catch (_) {}
        });
      }
      if (!data) idx++;
    }
    return data;
  };
}

async

@Raynos can you please explain what kind of async streaming is missing?

function(){
  return function*(){
    return yield fs.readFile(nextLog)
  }
}

abort

@Raynos can you explain what you mean by "missing abort"?

We have ending streams by calling yield fn(true)l

use `yield new Error()` instead of `throw new Error()`

the former allows you to pass errors to generator runner without going throw the exception system.

This works cleanly with raynos/gens and Agen spec

write co-from-stream

add benchmarks

excellent idea by @dominictarr.

compare with:

• node core stream 2
• node core stream 1
• pull stream
• pull stream without libs
• minstream
• minstream without libs
• reducers

add passthrough example

write co-to-stream

error should end all streams

what about when you *need* a parallel operation

say, a merge stream,
that takes N sorted streams and merges them into one stream?
(it would pull from each stream, but output chunks in correct order -
for example, to do a merge sort, or LSMT compaction)

A simpler one would just to merge multiple streams into one, but not control the order.

parallel map that doesn't wait for data to finish before continuing (out of order)

Looking to do the following:

1. pass in a bunch of file paths
2. map over each file to read it's contents
3. attach meta data to each file as they finish (out of order)
4. collect the files

Basically wondering how to pull off step 3, without waiting for all the contents to be read in the map