loop over an array in series?

[billiegoose]

loop over an array in series?

[billiegoose]

It took being exposed a few more times before the generator ideas really clicked, but now I don't think I can go back! Closing this because I can. Thanks again for such an awesome library.

[getify]

if what you're getting at is that you need a lazy sequence handling (because you can't process the entire array at once), there's a couple of options:

1. use a generator with runner(..):

   ASQ()
   .runner(function*(){
     var v;
     while (v = data.shift()) {
        // do stuff for each item in data
        // including `yield` pauses for asynchrony
     }
   });

2. use a reactive sequence to model a stream of data:

   var data = ASQ.react.of("one","two","three");
   
   data.val(function(v){
     // process `v` one at a time
   });

   With the reactive sequence as shown, the .val(..) call (and any other chain you specify there) will be repeated for each event message in the data reactive sequence (aka "observable").

Do either of those spark different ways of thinking about your problem? If you weren't using coffeescript (which I'm rather unfamiliar with), I might be able to see more nuance to your specific needs.

[billiegoose]

Thank you for replying so quickly! Sorry for the delay, I saw your reply late at night and forgot to respond.

1. looks almost exactly like what I am trying to do. However I haven't used generators in a project before, so I'm not sure how to weave them together with callbacks.
2. Possibly? But that seems to still run concurrently. I'd need to somehow pause the stream and unpause it after processing each item.

Just because you have a much keener eye for design patterns than I, here's a motivated example of how I've been using asynquence:

// OLD VERSION
function visitURL(url, callback) {
  var results = {}; // aggregate results of stuff
  results.url = url;
  browser.open(function(err, url){
    if (err) return callback(); // abort and proceed with next url on error
    // Do a bunch of things with page... type, click, etc
    // Pray the browser doesn't crash from memory leaks.
    browser.screenshot(function(err, image) {
      if (err) return callback(); // abort and proceed with next url on error
      results.image = image;
      database.save(results, function(err){
        // Done testing this url!
        callback()
      })
    })
  })
})

Re-written using ASQ, it becomes much tidier:

// NEW VERSION
function visitURL(url, callback) {
  var results = {}; // aggregate results of stuff
  results.url = url;
  ASQ(url)
  .then(function(done){
    browser.open(done.errfcb)
  })
  .then(function(done){
    browser.screenshot(done.errfcb)
  })
  .then(function(done, image){
    results.image = image
    database.save(results, done.errfcb)
  })
  .val(callback)
  .or(function(err){
    console.log(url, err)
    callback()
  })
})

Now I want to call visitURL for each URL in an array urls, but it has to be in sequence because I only have one "browser" object.

// Visit all the URLs
ASQ()
.until(function(done) {
  visitURL(urls.shift(), function() {
    if (urls.length > 0) {
      done.fail()
    } else {
      done()
    }
  })
})
.val(function(){
  console.log('Finished')
  process.exit()
});

This just feels odd. I feel like intuitively, I should be able to write something clean-looking like:

ASQ(urls)
.mapSeries(visitURL)
.val(function(){
  console.log('Finished')
  process.exit()
});

[getify]

First off, instead of having the browser.* calls in visitURL(..) use callbacks with then(..) calls, use wrap(..)ed versions of those utilities, like this:

var results = {};
results.url = url;

ASQ(url)
.seq( ASQ.wrap(browser.open) )
.seq( ASQ.wrap(browser.screenshot) )
.val(function(image){
   results.image = image;
   return results;
})
.seq( ASQ.wrap(database.save) );

Next, instead of having visitURL(..) take callback, have it just return the sequence (aka promise), like this:

function visitURL(url) {
   var results = {};
   results.url = url;

   return ASQ(url)
      .seq( ASQ.wrap(browser.open) )
      .seq( ASQ.wrap(browser.screenshot) )
      .val(function(image){
         results.image = image;
         return results;
      })
      .seq( ASQ.wrap(database.save) );
}

See how we return the sequence? Now, you can use visitURL(..) like this:

ASQ()
.seq.apply(null,
   urls.map(function(url){
      return function handleURL(){
         return visitURL(url);
      };
   })
)
.val(function(){
   console.log("Finished");
   process.exit();
});

Let me explain how that works. We take the urls array and do a map(..) on it, creating a new list of functions, each called handleURL(). Each of those functions calls visitURL(url) and returns the result, which is itself a sequence (aka promise), as we defined above. Now, we take that new list of functions (each which produces a sequence) and spread it out -- using apply(..) with null for the this arg -- as arguments to a seq(..) call.

seq(..) expects to either receive a sequence, or a function that will produce a sequence. If it receives multiple arguments, it processes them serially (not concurrently!!), moving onto the next entry in the arguments only once the previous one finishes.

IOW, passing multiple sequence-producing functions to seq(..) is the same thing as the mapSeries(..) you dreamed.

---

But, the goodness doesn't stop there! Instead of using all these sequence chains, let's use generators (far better readability, IMO).

First, convert visitURL(..) into a generator:

function *visitURL(url) {
   yield ASQ.wrap(browser.open)(url);
   var image = yield ASQ.wrap(browser.screenshot)();
   yield ASQ.wrap(database.save)({
      url: url,
      image: image
   });
}

See how much nicer that is!? The key magic to understand is that yield inside a generator (when driven by an intelligent runner like ASQ#runner(..)) pauses the local code so it looks sync even though it's actually waiting on async stuff to resume (like the browser.open(..) call, etc). Our sequence chain from before now just becomes a series of sync-looking statements with yield in each one. Cool, huh!?

Now, here's how we use our new *visitURL(..) generator: we convert our previous approach -- the array-map pushed in as seq(..) arguments -- into a simple while loop inside another generator, with yield* delegation to the *visitURL(..) generator each iteration.

ASQ()
.runner(function*(){
   var url;
   while (url = urls.shift()) {
      yield *visitURL(url);
   }
   console.log("Finished");
   process.exit();
});

Bam! I dunno about you, but I think these generator versions are waaaaay better and more readable than the sequence chain version. :)

[billiegoose]

Wow! Just wow!

ASQ()
.runner(function*(){
   var url;
   while (url = urls.shift()) {
      yield *visitURL(url);
   }
   console.log("Finished");
   process.exit();
});

That is far cooler/elegant than what I've been doing. But it would require re-writing everything to use yield. Which I now really want to do... and yet... (sigh) my existing code works. I'll resist the urge to refactor.

Regarding the first suggestion:

ASQ()
.seq.apply(null,
   urls.map(function(url){
      return function handleURL(){
         return visitURL(url);
      };
   })
)
.val(function(){
   console.log("Finished");
   process.exit();
});

Mixed feelings. a) It's absolutely amazing. b) It doesn't save a lot of typing, or even seem as intuitive as the .until() loop approach. c) The seq.apply(null, _) bit seems hacky? I mean, why doesn't .seq just take an array. Although now I'm curious whether it would work with ES6 spread operator...

.seq(...urls.map(function(url){

[getify]

it would require re-writing everything to use yield.

Why? I'd recommend refactoring little by little. No reason that you can't improve pieces and layers one at a time, I would think. Any improvement is better than propagating code in older, harder to understand styles. That kind of code is the first to get thrown away in future rewrites anyway. :)

Besides, having even one piece of your code written in a better style is a small chance that any new code will follow that pattern instead of the old stuff.

why doesn't .seq just take an array.

Because all the methods (like seq(..), val(..), then(..), etc) also take values as arguments, instead of having to send a function that returns the value immediately. As such, there'd be no way to know if you wanted an array to be a single passed-through value or spread out.

Although now I'm curious whether it would work with ES6 spread operator...

yes, of course. Especially since we can use the ..., there's no reason that seq(..) should auto-spread the array... better to force the code to be explicit about spread or not. :)

But I just didn't want to muddy the discussion with too much ES6 sugar that not everyone is aware of. Also notice I didn't dribble arrow functions all over, as many are accustomed to. :)