Comments (7)
Could you provide an example?
from scraperjs.
For example on this page there is a 'load more' button to get all the events to show in the DOM before I can scrape them. So I've set up an interval to check the status of the load more button, and click it if needed like so
if ($('.load_more_link').length > 0) {
var clicker = setInterval(function() {
$('.load_more_link').click()
if ($('.load_more_link').length == 0) {
clearInterval(clicker) // done here
};
}, 1000)
}
Then once it's done, I want to return the total number of events along with some other info... but I don't see how I can return any info once the function becomes asynchronous.
from scraperjs.
This is a common pattern.
phantomjs does have the functionality to trigger events after a page has been loaded. Maybe we can add it into ScraperPromise.js
? so that we have something like:
var scraperjs = require('scraperjs');
scraperjs.DynamicScraper.create('https://news.ycombinator.com/')
.triggerEvent(function(){
//trigger events
});
But, since phantomjs is sandboxed, we need to think of a way to tell the promise that event has finished. set a waiting time is one way (poor one).
from scraperjs.
@ksloan , for that kind of sites I find that it is better just to inspect the ajax calls and work from there. You could also try to to use the delay
promise between two scrape
promises.
@cjackie , I might look into that sometime in the future.
from scraperjs.
The v0.3.0 has an async
promise, you can see the promise which allows to check for events and then trigger them.
from scraperjs.
Amazing!! Thank you!
from scraperjs.
Can you provide example using aync?
from scraperjs.
Related Issues (20)
- Using iconv-lite to decode ISO-8859-1 page HOT 1
- Receiving 'cannot read error property of null' when using the dynamic scraper. HOT 1
- Scraper function will not work unless it matches regex pattern
- Does Dynamic Scraper work with Node version 4.0.0 ?
- Chaining promises
- Character encoding - possible to set it? HOT 2
- error Exceeded maxRedirects HOT 1
- Is it possible to run scraperjs through php cron? HOT 3
- Cannot inject JS functions HOT 1
- Pass parameter to scrape evaluate function HOT 1
- DynamicScraper can not run HOT 5
- How to scrape using html files if the site did not declare any "class"
- Unable to perform inside loop? HOT 1
- Not an issue, but I didn't know where to put it
- Error: Can't find variable: $
- Maintainer?
- why i am getting saved as false when i run script using PHP
- Scrape local file
- how to use it in Angular
- Security Vulnerabilities HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scraperjs.