phantombuster / nickjs Goto Github PK
View Code? Open in Web Editor NEWWeb scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Home Page: https://nickjs.org
License: ISC License
Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Home Page: https://nickjs.org
License: ISC License
Nickjs: 0.3.6
Chrome: 67.0.3396.62
Ubuntu: 16.04.4
I am trying to crawl google map and got following error.
2018-06-01 10:34 : CHROME STDERR: [0601/103455.634767:ERROR:gpu_process_transport_factory.cc(1017)] Lost UI shared context.
2018-06-01 10:34 : CHROME STDERR:
Best regards,
Environemnt:
Ubuntu 16.04
Nickjs ^0.3.6
Nodejs v8.11.1
Hello there,
I am getting this even I use the tab.close().
Do you know why I am getting more chrome process?
Should I use nick.exit() after the execution if I want to kill all chrome process?
Regards,
Tabs
i think there is a lot to be learned from cypress
https://www.cypress.io/
it just isn't made to do scrapping.
> Tab 1: Page JavaScript error: ReferenceError: c is not defined
at sendLog (https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/plugins/every_cookie_mac_82990d4.js:4:177)
at https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/plugins/every_cookie_mac_82990d4.js:5:127
> Tab 1: Navigation (open): https://www.baidu.com/
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=he&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=2&pwd=h&cb=jQuery1102022635599116432248_1531837205312&_=1531837205315:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=hea&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=3&pwd=he&cb=jQuery1102022635599116432248_1531837205312&_=1531837205316:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=h&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=1&cb=jQuery1102022635599116432248_1531837205312&_=1531837205314:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headle&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=6&pwd=headl&cb=jQuery1102022635599116432248_1531837205312&_=1531837205319:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=9&pwd=headless&cb=jQuery1102022635599116432248_1531837205312&_=1531837205322:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headl&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=5&pwd=head&cb=jQuery1102022635599116432248_1531837205312&_=1531837205318:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=8&pwd=headles&cb=jQuery1102022635599116432248_1531837205312&_=1531837205321:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=head&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=4&pwd=hea&cb=jQuery1102022635599116432248_1531837205312&_=1531837205317:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headles&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=7&pwd=headle&cb=jQuery1102022635599116432248_1531837205312&_=1531837205320:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20c&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=10&pwd=headless%20&cb=jQuery1102022635599116432248_1531837205312&_=1531837205323:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20ch&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=11&pwd=headless%20c&cb=jQuery1102022635599116432248_1531837205312&_=1531837205324:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20chro&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=13&pwd=headless%20chr&cb=jQuery1102022635599116432248_1531837205312&_=1531837205326:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20chr&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=12&pwd=headless%20ch&cb=jQuery1102022635599116432248_1531837205312&_=1531837205325:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20chrom&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=14&pwd=headless%20chro&cb=jQuery1102022635599116432248_1531837205312&_=1531837205327:1:1
> Tab 1: Page JavaScript error: TypeError: jQuery1102022635599116432248_1531837205312 is not a function
at https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?wd=headless%20chrome&json=1&p=3&sid=1423_21122_26350_26810&req=2&csor=15&pwd=headless%20chrom&cb=jQuery1102022635599116432248_1531837205312&_=1531837205328:1:1
I am making web based api with express4 and pm2.
When I call nick.exit(), my api auto-restarts because it is calling exit() function.
So, is there any way that I can finish using nickjs? like not auto-restarting the api?
How to run on GCF? getting this error -
could not start chrome: Error: spawn google-chrome-beta ENOENT
I'm not sure if one or both of these options is being ignored:
$ node build/test.js
tab.open()
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-content/plugins/swiftype-search/assets/install_swiftype.min.js?ver=4.8.2
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-includes/js/jquery/jquery.js?ver=1.12.4
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1
...
test.js:
Nick = require('nickjs');
nick = new Nick(options = {
printNavigation: false,
printResourceErrors: false,
printPageErrors: false,
blacklist: [/.*\.js.*/]
});
(async() => {
var tab;
tab = (await nick.newTab());
console.log('tab.open()');
return tab.open('https://mixergy.com/interviews/');
})();
Is there a possible future support for mouse simulations?
How do you insert a text inside a textarea that is not belongs to a form tag?
hi , when i run the demo, it appears Error: getaddrinfo ENOTFOUND localhost undefined:9222
In ubuntu Server, there is not chrome instead of chromium. Is it the same way to set the chromium path to Chrome_Path environment variable?
how solvethis error :
when run with thiscommand
./node_modules/casperjs/bin/casperjs lib/myNewBot.js
With Node 6.11.3 LTS and Node 8.5.0
Chrome Version 61.0.3163.91 (Official Build) (64-bit)
Using sample on https://nickjs.org/
It always say: "Error: Could not start chrome: Error: spawn google-chrome-unstable ENOENT"
Same environment, I run puppeteer normally
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch();
console.log(await browser.version());
browser.close();
})();
Result is HeadlessChrome/62.0.3198.0
Hello there,
I would like to change userAgent after initialization of Ncickjs.
What I found is following.
tab = await nick.newTab();
tab.driver.client.Network.setUserAgentOverride({"userAgent":userAgent});
Is this correct way to do?
regards,
Is there a way to recover or instantiate a tab again when it crashed? Whats the value of the tab when it crashed?
Google Chrome: 66.0.3359.117
Nickjs: 0.3.6
Error:
2018-05-15 11:17 : CHROME STDERR: [0515/111735.657188:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111735.660133:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111745.945705:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111745.948669:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111748.365746:WARNING:spdy_session.cc(2991)] Received HEADERS for invalid stream 767
Do you know how to solve this issue?
I get this :
Error: timeout: load event did not fire after xxx ms
while I'm trying to open website such as azlyrics.com or genius.com, even when I extend the timeout option to 60000ms.
I can quickly connect to https://news.ycombinator.com/ or other websites though. Are some websites preventing headless browser connections ? If so, is there any workaround ?
My NickJS script was working fine a few weeks ago. I don't think I changed anything, but it stopped working. I tried updating from NickJS 2.0 to 2.3 and Chrome dev to the latest version with no improvement.
Apparently it is hanging at the call to tab.open(url)
. I also tried running the sample code at https://nickjs.org/, and it hangs at the same method. How do I figure out why NickJS is hanging there? (The PhantomBuster demo still works, so I think it must be something with my environment)
Environment:
Simple code to reproduce (CoffeeScript):
Note the callback is never called.
Nick = require 'nickjs'
nick = new Nick()
callback = (err, httpCode, httpStatus, url) =>
console.log 'Callback:'
console.log err
( () =>
tab = await nick.newTab()
console.log 'tab.open()'
tab.open 'https://mixergy.com/interviews/', callback
)()
Simple code to reproduce (JS):
Note the callback is never called.
var Nick, callback, nick;
Nick = require('nickjs');
nick = new Nick();
callback = (err, httpCode, httpStatus, url) => {
console.log('Callback:');
return console.log(err);
};
(async() => {
var tab;
tab = (await nick.newTab());
console.log('tab.open()');
return tab.open('https://mixergy.com/interviews/', callback);
})();
I want to use casper + phantom, not headless chrome.
Then, I have following source code.
require('babel-polyfill')
const Nick = require('nickjs')
const Promise = require('bluebird')
const nick = new Nick()
nick.newTab().then(async function(tab) {
await tab.open("news.ycombinator.com")
await tab.untilVisible("#hnmain") // Make sure we have loaded the page
await tab.inject("/home/vagrant/test/nickjs_casper/src/js/jquery-3.3.1.slim.min.js") // We're going to use jQuery to scrape
const hackerNewsLinks = await tab.evaluate((arg, callback) => {
// Here we're in the page context. It's like being in your browser's inspector tool
const data = []
$(".athing").each((index, element) => { data.push({
title: $(element).find(".storylink").text(),
url: $(element).find(".storylink").attr("href")
})
})
callback(null, data)
})
console.log(JSON.stringify(hackerNewsLinks, null, 2))
})
.then(() => nick.exit())
.catch((err) => {
console.log('Oops, an error occurred: ' + err)
nick.exit(1)
})
But, I am getting the following error.
Oops, an error occurred: Error: in evaluated code (initial call): EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://cdnjs.cloudflare.com/".
Do you know why I am getting this error?
Does the evaluate function include all the execution context in the page? Like the elements in an iframe
Hello there,
Possibly, this is bug on headless chrome, but, I want to confirm if this is a bug.
When I crawl like following on google map, the result is less than I use just chrome browser.
Open this URL.
https://www.google.co.jp/maps/@35.6603142,139.7055783,21z
sendkey like shibuya web consultant
The result of google chrome has more than 20 search results.
On the other hand, the result of chrome headless browser has aorund 8 search results.
Do you have the same result?
My environment is as following.
ubuntu 18.04
Google Chrome 67.0.3396.99
NickJs the latest one
Regards,
This error occurs after upgrading to latest version of Chrome:
error when initializing new chrome tab: Error: 'Network.setRequestInterceptionEnabled' wasn't found
Confirmed with these release channels of Chrome:
Environment:
The code:
Nick = require 'nickjs'
nick = new Nick()
nick.newTab().then (tab) =>
await console.log "Testing..."
.then () =>
console.log 'Completed with no errors!'
nick.exit 0
.catch (err) =>
console.log 'Something went wrong:'
console.log err
nick.exit 1
Please enable changing NickJS options after the initial constructor. Specifically, I would like to modify:
loadImages
flagprint*
flags would be nice, too)Motivation: I would like to use a special configuration just when logging in
loadImages=true
and no blacklistsloadImages=false
and blacklists (and load faster without images & JS files)Need some help getting Chrome running in different environments.
In a different thread, you said nickjs looks for google-chrome-unstable before checking CHROME_PATH in environment variables.
Is using google-chrome-unstable the best way to have the same code execute in both a local (Mac) and remote (EC2/Elastic Beanstalk) node.js setup?
If so, can you help with download/install instructions of google-chrome-unstable?
If not, can you provide help with getting nickjs to recognize Chrome in an EC2 instance?
Thx - I know these are probably noob questions.
could not connect to chrome debugger after 24 tries (10s): Error: connect ECONNREFUSED 127.0.0.1:9222
this just popped up for some reason on heroku, i have legacy dynos running the same code and working perfectly. tried to add a fresh dyno and getting this error for the first time
settings are:
Stack: heroku-16
node: 8.9.0
npm: 5.5.1
How is scaling supposed to work with nickjs?
Does instantiating a new nickjs starts a new session (suppose you want to keep tabs independent from each other)
Hello guys!
You've been doing a great job with NickJS library!
But working with NickJs on NodeJs I had the following server error: "Chrome subprocess killed by signal SIGTERM". In fact, Chrome is killed by SIGTERM kernel process while NickJs tabs are evaluating.
It seems the problem occurs when code open/close tabs for long time, in particular when pages have a lot of content and scripts.
Test script is written to reflect the problem on the real project, where pages also run simultaneously in different tabs for long time. During the work CPU and Memory consumption in Task Manager looks fine, so killing the Chrome process is very unexpected.
My environment: NodeJS v8.8.1, NickJs + Desktop Chrome 63.0.3239.108 (Official Build) (64-bit).
Additional packages: Bluebird - for advanced Promise usage.
OS: Windows 8 (But the same error occurs on Ubuntu 16.04)
Here is my test file 'pureNickTest.js':
const Promise = require('bluebird');
const nick = new NickJs();
const testUrl = "http://edition.cnn.com/2017/12/15/politics/next-alabama-states/index.html";
const runTab = async (url) => {
const tab = await nick.newTab();
await tab.open(url);
await tab.untilVisible("body");
await tab.inject(`${__dirname}/libs/jquery-3.2.1.min.js`);
await tab.wait(3000);
await tab.scrollToBottom();
const results = await tab.evaluate((arg, callback)=>{
const blocks = $("body").find('div');
callback(null, blocks.length);
});
await tab.close();
};
const runTestTabs = () => {
const fakeArray = new Array(100);
return Promise.each(fakeArray, () => {
return runTab(testUrl);
});
}
(async ()=> {
await runTestTabs();
console.log("Test complete!");
nick.exit();
})();
Equivalent code for puppeteer works fine and doesn't crush with this error.
I would appreciate any help on this issue!
Hi guys,
You made a great tool, but I'm running in an issue.
when I try the sample code in node.js when the code is reaching nick.exit(0); the whole node process is killed I was making some tests with postman and as a result the response from the server is always : "Could not get any response" but in the node logs I have :
"server started on: 3000"
"Tab 1: Navigation (open): http://:www.google.com"
"Process finished with exit code 0"
and after the node server is not responding and is killed.
I tried also on a remote server the node server is restarting but every time the process is killed.
For now I found a workaround "tab.close()" but it looks like the chrome is staying there in background and every time a new Nick is created a new chrome is launched.
Can you give me some advice about this or is it a bug ?
I am trying to use a SOCKS5 proxy to connect and "http://" is getting appended to the front. The SOCKS5 proxy url takes this form "socks5://username:password@domain:1080".
I tried running NickJs in node9-alpine image and I get this error:
CHROME STDERR: [0204/101402.974481:WARNING:dns_config_service_posix.cc(326)] Failed to read DnsConfig.
CHROME STDERR: [0204/101402.984254:ERROR:devtools_http_handler.cc(759)] Error writing DevTools active port to file
> It took 105ms to start and connect to Chrome (1 tries)
{
"level": "error",
"message": "cannot connect to chrome tab: Error: Unknown command: protocol"
}
I have added the chromium browser to the image and set the CHROME_PATH
to the chromium-browser. It is also present there, I checked with running a bash in the container.
Does anyone know what to do?
My Dockerfile:
FROM node:9-alpine
ENV CHROME_PATH=/usr/bin/chromium-browser
ENV NICKJS_NO_SANDBOX=1
# ╒═════════════════════════
# │ Copy from builder
# ╘═════════════════════════
RUN mkdir -p /app
WORKDIR /app
# Node modules
COPY --from=builder /app-prod/node_modules ./node_modules
# App files
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/res ./res
COPY --from=builder /app/package.json .
COPY --from=builder /app/util/healthcheck.js .
COPY --from=builder /app/start.sh .
RUN rm -f /app/dist/source.js /app/dist/source.js.map
# ╒═════════════════════════
# │ INSTALL NATIVE
# ╘═════════════════════════
# Install native app dependencies
RUN set -ex; \
apk add --no-cache \
rsync \
openssh \
udev \
ttf-freefont \
chromium
# ╒═════════════════════════
# │ SETUP HEALTH CHECK
# ╘═════════════════════════
HEALTHCHECK --interval=12s --timeout=12s --retries=3 --start-period=60s \
CMD node ./util/healthcheck.js
# Run the container under "node" user by default
#USER node
CMD [ "npm", "run", "serve" ]
How do i simulate a Enter event, like pressing enter after sending keys in a textarea?
What would be the best way to set a different proxy and user-agent on each tab open, testing this out currently but not sure if this is the right approach:
const nick = new Nick({
printNavigation: true,
printResourceErrors: true,
printPageErrors: false,
resourceTimeout: 10000,
})
// start loop logic
;(async () => {
nick.options.userAgent = randomUserAgent;
nick.options.httpProxy = randomProxy;
const tab = await nick.newTab()
await tab.open(url)
// Clean up
await nick.deleteAllCookies()
await tab.close()
})()
// end loop logic
Initially I had const nick = new Nick inside my loop logic and that crashed node after 100 tab opens, pretty sure why :)
I just paste npm install nickjs
and I received some warnings.
Shouldn't a chromium browser gets downloaded as part of nickJS installation instead of manually pointing chromium executable path?
Thanks for the library.
Hi and first thanks for NickJs and PhantomBuster
This is an Help Me Please issue.
Im'not very familiar with async/await function, so i'm practicing with this.
My question is:
How can i grab a bunch of urls (Like my example code)
And loop through an array to open new tab for each url, take a screenshot, close the tab, close nick.
I'm a user of Phantombuster too (in experimentation in my company) and i'm really really stuck with this.
Anyone to help me ?
import 'babel-polyfill'
import Nick from 'nickjs'
const nick = new Nick()
nick.newTab().then(async function (tab) {
await tab.open('https://news.ycombinator.com/')
await tab.waitUntilVisible('#hnmain')
await tab.inject('https://code.jquery.com/jquery-3.1.1.slim.min.js')
const urls = tab.evaluate((arg, callback) => {
const data = []
$('.athing').each((index, element) => {
data.push($(element).find('.storylink').attr('href'))
})
callback(null, data)
})
return urls
}).then((urls) => {
for (var i = 0; i < urls.length; i++) {
// here i want open a new tab for each url i have in my urls array
// And i want to perform a screenshot or a evaluate function
}
//and after that i want to quit nick
nick.exit()
})
.catch((err) => {
console.log('Oops, an error occurred: ' + err)
nick.exit(1)
})
Best Regards
Can nick run inside an existing running electron instance and use the api to control it?
If so, how would the configuration look like ?
I'm using NickJs browser emulation and am attempting to add a header when opening a page. I'm monitoring the request in Fiddler and see that the header is not being added.
How do I add the header?
Here's my current code:
await tab.open("https://www.google.com/", {headers: {"Accept-Language": "en-US,en;q=0.9"}})
First off, Nick is amazing. Thank you.
I tried upgrading to .0.3.6
(from .0.3.0
) get around the Chrome subprocess killed by signal SIGTERM
issues.
Now I get a Error: timeout: load event did not fire after 10002ms
on my code. Any ideas? The HN example still works so I'm not sure what's wrong with my code.
const Nick = require("nickjs");
const nick = new Nick();
(async () => {
const tab = await nick.newTab();
let urls = ["https://www.lawfwfewef.com", "https://www.awfwaffe.com"];
let outsideresults = [];
for (let url of urls) {
const insideresults = [];
await tab.open(url);
await tab.wait(9000);
await tab.untilVisible("html"); // Make sure we have loaded the page
} // for of loop
})()
.then(() => {
console.log("Job done!");
nick.exit();
})
.catch(err => {
console.log(`Something went wrong: ${err}`);
nick.exit(1);
});
Hi !
I begin using your lib and I'm wondering if and how I could use proxy to connect to the web ?
thanks for your work
Hugo
Hello there,
Is reset option working?
I found that it is not working and so the previous value is left in the input box.
Here is my code.
await tab.sendKeys('#username', id, {reset:true,keepForcus:false});
Can you confirm that?
OS: Ubuntu 16.04
NickJS: The latest one
Google Chrome: 67.0.3396.62
Best regards,
Is there a way to log/view the client request headers and the server response headers?
When I try to launch the example I get this error :
SyntaxError: Unexpected token '>'
phantomjs://code/test.js:4 in injectJs
phantomjs://code/bootstrap.js:456
To launch the script I use :
node_modules/casperjs/bin/casperjs test.js
test.js contains the example from readme.md
EDIT :
I just discoered that casper.js or phantom is not suporting ES6
setTimeout( () => console.log('Hello'), 3000 );
this fail :'(
#What I've tried
*Reinstalling Chrome
*Modifying my installation of Nickjs to try enabling downloads
*Reinstalling a clean Nickjs installation
#Expected behavior
Downloadable files will be downloaded when navigated to
#Actual behavior
Downloadable files are not downloaded
Hi, it's me Again.
It sound dumb (and not really relative to nick) but how can i use Node API like fs (for writing my scrapping results in a json file) ?
Here my code
require('babel-polyfill')
const Nick = require('nickjs')
const nick = new Nick()
var fs = require('fs');
nick.newTab().then(async function (tab) {
await tab.open('http://example.com')
await tab.waitUntilVisible('.products-grid')
await tab.inject('./jquery.js')
const urls = await tab.evaluate((arg, callback) => {
const urls = []
$("li.item").each((index, element) => {
urls.push($(element).find("div.product-content-wrapper").find('h3.product-name').find('a').attr('href'))
})
callback(null, urls)
})
var data = []
for (var i = 0; i < urls.length; i++) {
await tab.open(urls[i])
await tab.waitUntilVisible('.catalog-product-view')
await tab.inject('./jquery.js')
const type = await tab.evaluate((arg, callback) => {
var type = $('div.product-name').find('h1').text()
callback(null, type)
})
const img = await tab.evaluate((arg, callback) => {
var img = $('div#map_container').find('img').attr('src')
callback(null, img)
})
data.push({ type: type, img: img })
await tab.screenshot('./uploads/' + i + '.png')
}
fs.writeFile('./myjsonfile.json', data, 'utf8', ()=>{
nick.exit()
});
}).catch((err) => {
console.log('Oops, an error occurred: ' + err)
nick.exit(1)
})
Here at the end I want to write a json file with my results (var data is an Object).
But in the execution i have an error
undefined is not a function (evaluating 'fs.writeFile('./myjson.json', data, function () {
})')
For my defense, i'm really new to PhantomJs/CasperJs and Babel.
I know my script is executed in a CasperJS context, but i can't figured out why i can't use fs (even with the require('babel-polyfill')
Anyone to help me learn and make me a better dev ?
Hi,
thank you for your time in advance.
I am having login issue with nickjs.
I took the cookies from my browser and I set it up before opening the URL.
This is my code. As you can see from the screenshot, it fills the email and the password (the "name" attributes are the same with the input's keys) but it stops there (you can see my shell console output bellow).
I tried to navigate to a url which requires log-in, but it gives me the login form again, which means the login process wasn't successfull.
Would you help me to solve this issue? I would appreciate your help.
Regards,
Manol.
`
const selector = "form[name='signIn']";
const inputs = {
'email': "real email",
'password': "real password"
};
try {
await tab.waitUntilVisible(selector, 5000)
await tab.fill(selector, inputs, { submit: true })
//await tab.click("input#signInSubmit")
console.log("Form sent!")
await tab.screenshot('./login.png')
} catch (err) {
console.log("Form not found:", err)
}`
`root@06131ed62a3b:/app# node index.js
Fatal: Chrome subprocess exited with code 1
Tab 1: Navigation (open): https://www.amazon.com/ap/signin?_encoding=UTF8&accountStatusPolicy=P1&openid.assoc_handle=usflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fcss%2Forder-history%3Fie%3DUTF8%26ref_%3Dnav_youraccount_orders&pageId=webcs-yourorder&showRmrMe=1
Tab 1: Navigation (formSubmissionPost): https://www.amazon.com/ap/signin
Form sent!
Job done!
`
As the following code, last console.log should be printed but the app would be forced exit as nick.exit() called. By remove nick.exit() line, console.log print the result but the app would not end as it should be. (I will have to Ctrl-c to terminate)
const nick = new Nick()
let tab = await nick.newTab()
await tab.open(url)
await tab.untilVisible('.tbl')
let content = await tab.getContent()
nick.exit()
console.log('This line should be printed with content', content)
Hi.
Do you have an image to dockerize nickjs (latest version with chrome) ?
I tried to dockerize my app with a very simple dockerfile with the latest node image
FROM node:latest
ADD ./app /app
WORKDIR /app
ENV CHROME_PATH=/usr/bin/google-chrome
RUN apt-get update && install google-chrome-stable
RUN npm install
CMD npm start
My app it's your example with ycombinator and nothing more.
The google-chrome-stable version is 62 (in accordance with your documentation).
The image build is ok, chrome's installation is ok path is ok (Check with which google-crome
and echo "$CHROME_PATH"
).
But when i run a new container or when i run node index.js
to launch my node script inside a new container i got this error:
Fatal: Chrome subprocess exited with code 1
And i can't figured out what is my problem.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.