Using osmosis with multiple sites in parallel

[rchipka]

Using osmosis with multiple sites in parallel

[ruimgoncalves]

Here is some sample code demonstrating the problem
Toggle experiment value to see different results
In my case I cant get the results in parallel from the 2 sites, the problem gets worst if you use post requests.
Is there any workaround for this?
Thanks

experiment = true

setTimeout ()->
  osm = require("osmosis")
  osm
    .get 'www.npmjs.com/search?q=scraper'
    .set
      query : 'input#site-search@value'
      packages : [
          osm.find 'ul.search-results li'
            .set
              name : '.package-details h3 a.name'
              description : '.package-details p.description'
      ]
    .data (listing)->
      console.log listing
    .error (info)->
      console.error "erro npmjs", info
, if experiment then 10 else 0

setTimeout ()->
  osm = require("osmosis")
  osm
    .get 'www.craigslist.org/about/sites'
    .set
      'location' : ['h1 + div a']
    .data (listing)->
      console.log listing.location.length
    .error (info)->
      console.error "erro craigslist", info
, if experiment then 0 else 10

[cezary]

Pretty sure I'm running into this same problem; parsing multiple sites concurrently with osmosis, only the last calls done.

[andineck]

I'm experiencing the same problem, but have no Idea how to fix it... Anyone?

[rchipka]

@cezary this is a known behavior of Osmosis. Osmosis only keeps a global stack count, so only the last instance will call done. There's no bulletproof way of implementing per-instance stack counting because there are many scenarios where a "done" instance can start back up again. The only way to really know is when the actual Document objects are being garbage collected. I've been adding features to libxmljs that will allow osmosis to have much more functionality. One of my next libxmljs pull requests will probably be a callback function for the Document destructor. This will allow for accurate per-instance stack counting and done calling.

[rchipka]

I'll be looking into this issue as soon as I can.

[andineck]

@rc0x03 👍 thank you!

[rchipka]

This should be fixed in the latest version of Osmosis