joelgriffith / navalia Goto Github PK
View Code? Open in Web Editor NEWA bullet-proof, fast, and reliable headless browser API
Home Page: https://joelgriffith.github.io/navalia/
License: GNU General Public License v3.0
A bullet-proof, fast, and reliable headless browser API
Home Page: https://joelgriffith.github.io/navalia/
License: GNU General Public License v3.0
Hi there,
I have started to implement Navalia, it works great on production but fails when trying on localhost.
Here is my test case:
const { Chrome } = require('navalia');
const pageUrl = 'http://localhost:3000';
describe('Homepage', () => {
let chrome = {};
// Setup a clean instance for each test
beforeEach(() => {
chrome = new Chrome({
timeout: 100000,
flags: { headless: false }
});
});
// Tear down for each test
afterEach(() => {
return chrome.done();
});
it('should load', () => {
return chrome.goto(pageUrl)
.then(() => chrome.exists('[data-test="username"]'))
.then((exists) => expect(exists).toEqual(true));
});
});
When I use my production URL instead of localhost one it works fine. But with localhost I can see chrome opening, the page loads and the close before testing. Here are the logs
navalia:chrome :then() > Executing 1 actions +0ms
navalia:chrome :getChromeCDP() > starting chrome +2ms
navalia:chrome :getChromeCDP() > chrome launched on port 57985 +1s
navalia:chrome :goto() > going to http://localhost:3000 +0ms
navalia:chrome :goto() > waiting for pageload on http://localhost:3000 +2s
navalia:chrome :done() > finished +2s
navalia:chrome :done() > closing chrome +0ms
Not sure how to debug that?
Thanks,
Hi there, and thanks for this amazing library!
I'm trying to resize the browser window in order to capture automated full page screenshots for various screen sizes, but I'm seeing an issue where screenshots are always blank after resizing the browser window.
I'm using navalia 1.2.0 on mac os 10.12.06 with chrome 61.0.3163.100
I've created a minimal reproduction case here:
const { Chrome } = require('navalia')
const chrome = new Chrome()
const fs = require('fs')
chrome
.goto('https://google.com')
.then(() => chrome.screenshot('body'))
.then((img) => fs.writeFileSync('yep.png', img))
.then(() => chrome.size(640,640))
.then(() => chrome.screenshot('body'))
.then((img) => fs.writeFileSync('nope.png', img))
.then(() => chrome.done())
And here is yep.png
:
and nope.png
:
I'd be happy to submit a PR to fix this, but am at a bit of a loss as to how to debug since I'm not sure how to get some visibility into the execution of the chromedriver process. Any tips there would be much appreciated!
This example:
const { Chrome } = require('navalia');
const chrome = new Chrome();
chrome
.goto('https://amazon.com')
.type('input', 'Kindle')
.click('.buy-now')
.end()
.then(responses => {
console.log(responses); // ['https://www.amazon.com/', true, true, true]
});
causes the following error:
(node:13803) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Goto failed to load in the timeout specified
(node:13803) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
no way to work around it. Can't use navalia at all. Please help!
For example, when entering text into a search I'd like to press enter afterwards. Sometimes there's no button to press to confirm a text entry.
When trying to change the value of a dropdown using select chrome.select returns ReferenceError: option is not defined
async function test() {
await chrome.goto('http://localhost:3000/pva');
const x = await chrome.exists('#accountViewSelector') //true
const y = await chrome.html('#accountViewSelector') //returns correct dropdown <select ..
const z = await chrome.select('#accountViewSelector', '0'); // ReferenceError: option is not defined
chrome.done();
}
test();
Sole component being rendered
const PortfolioValueAnalysis = () => {
return (
<select id='accountViewSelector' className="form-control">
<option value='0'>0</option>
<option value='1'>1</option>
<option value='2'>2</option>
</select>
)
}
export default PortfolioValueAnalysis;
Love this repo and switching all our E2E tests to using Navalia!
Bumped into this issue with screenshot, my situation is the same as this closed issue: #63
However no matter how large I resize the browser to (I tried chrome.size(1600,900), chrome.size(800,800), chrome.size(2500, 1200), etc), once chrome.size is called, the screenshot taken is blank.
Also, it's interesting that the png generated is four times as large as the dimension given. For example, if I call chrome.size(1600, 900), the generated blank png is of dimension (3200, 1800).
Chrome just released code-coverage stuff in there devtools. I can see this being extremely useful in CI scenarios where consumers want to assert usage thresholds.
I've been evaluating navalia for integration testing on a new project. I really like the api and found it very easy to get it up and running, so great job!
I have a basic test that just makes sure our React app has actually loaded:
const { Chrome } = require('navalia')
const testUrl = process.env.TEST_URL || 'http://localhost:3000'
console.log(`TEST_URL: ${testUrl}`)
describe('Entry App', () => {
let chrome = {}
beforeEach(() => {
chrome = new Chrome()
})
afterEach(() => chrome.done())
it('should load entry page', async () => {
await chrome.goto(testUrl)
await chrome.wait(500)
expect(await chrome.exists('#root')).toBe(true)
}, 30000)
})
I'm having an issue where the test runs fine on my machine (macOS Sierra 10.12.5, Chrome 59.0.3071.115) but it's not working on our CI server. The CI is running in a docker container (RHEL7, Chrome 59.0.3071.115) and the error it shows is:
connect ECONNREFUSED 127.0.0.1:38502
at Object.exports._errnoException (util.js:1022:11)
at exports._exceptionWithHostPort (util.js:1045:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:14)
Interestingly, the test is also failing on another dev's machine (macOS Sierra 10.12.5, Chrome 60.something - don't have the exact version on hand) with a similar error but instead of 127.0.0.1
it has ::1
(IPv6).
So far the only machine it works on is mine (we have only tried these 3 but will be seeking others tomorrow to test), but I have no idea what I've done to be so special.
Any insights you may have are very much appreciated.
Hi,
I am trying to figure out how to even run this application. I know it runs on headless chrome but I have no idea on how to actually run the test script(s) that I have made.
Thanks!
I am using Chrome stable v60 (not Canary). The code is below
import { Chrome } from 'navalia';
const chrome = new Chrome();
async function screenshot() {
await chrome.goto('https://www.google.com');
await chrome.screenshot('body');
chrome.done();
}
screenshot();
I ran fine without error but no screenshot file either.
Any help?
Awesome project!
Does it make sense to support starting Chrome via Docker? It would allow us to use navalia
to start the same Chrome rendering on both our local machines (using MacOS) and our integration environment (Travis + Linux).
Getting this error : Unhandled promise rejection (rejection id: 1): Error: unexpected server response (500) and sometimes
Error: 'Target.createBrowserContext' wasn't found
Using Chrome Version 59.0.3071.115
Just tried this example:
chrome
.start()
.then((tab) =>
tab.navigate('https://www.google.com/')
.then(() => tab.screenshot('google.png'))
.then(() => tab.done())
);
Also tried other examples , getting the same 500 error.
A quick help would be appreciable.
When I use await chrome.wait(3000);
the test crashes with the code
Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
the error persists even if I put the parameter timeout...
await chrome.wait(3000, 5000);
await chrome.wait(3000, { timeout: 5000 });
chrome = new Chrome({ timeout: 5000 });
Hi! Awesome work, and inspiring too!
Any plans to support API testing, specially JSON responses?
Thanks again!
It seems a little heavy (and opinionated) to achieve what Nightmare.js does.
Really like this idea. Some thoughts I have:
Hi !
Thanks for this tool ๐
Just requesting a feature: add a way to pass custom headers to goto
.
Not sure how it works, but there's something about headers
in Network.Request
Here's the code I'm trying to run, compiled from two different examples in the docs:
const { Chrome } = require('navalia');
const chrome = new Chrome({chromeOptions: {
flags: {
headless: false
}
}});
chrome
.goto('https://amazon.com')
.type('input', 'Kindle')
.click('.buy-now')
.then((responses) => {
console.log(responses); // ['https://www.amazon.com/', true, true, true]
});
But when I run it it works as normal, like it did before I added the noheadless arg. Anything obvious I'm doing wrong?
Hi,
I would really like to be able to control the mobile prop of the captureScreenshot call.
Hey folks,
Looking to dust off this project a bit. Wondering if we should can the GraphQL interface? I'm thinking about a separate project for that since it seems to be not directly related to offering a decent API.
Thoughts?
Love this repo! I'm thinking about switching over one of my projects from NightmareJS to this but I need to use authenticated proxies. I know with the chrome command line flag you can pass in a proxy via --proxy-server=PROXY_URL_HERE
. However, this only works for non-authenticated proxies. In NightmareJS they handle this by listening for an authentication event and then logging in.
How could I implement something similar for this repo?
If you can provide me some guidance, I'd be willing to implement the PR.
Hi, thanks for the great work.
I'm trying to test some elements that have a onMouseDown
function bound to it with the Navalia click
method, but the event is not triggered.
Don't know if the best option is to create a new Navalia method called mouseDown
or make click
work with onClick
, onMouseDown
and other types of events like touch.
Currently, exists
will wait up to 10 seconds for the element to appear. If it's not there, the timer runs out and Navalia will throw. When a timer is being utilized, exists
should "catch" the timeout error and return false, which is the spirit of this API (true
when present, false
if not).
How do I access to Emulation.setVisibleSize
to set the window size before taking screenshot?
I have a scenario that I would like to assert the current URL, today I could not find how to do this.
Is there any way today using navalia.js to do this?
I actually came across this last week or so and couldn't really figure out what it was. Just now I found out that it's like chromeless (which is like Nightmare.js), except that it's actually feature complete and not just an empty v1.0
"Navalia" doesn't communicate anything to me and makes me think of some frontend framework, like Aurelia or something.
I'm learning navalia, and have a test that looks on the page for a certain element, like this:
.then(() => chrome.exists('[#ns-leftdrawer-list-item--atlas]'))
it succeeds, and my test then attempts to click on a child element, like this:
chrome.click('[#ns-leftdrawer-list-item--atlas > a]')
this fails, and the chatter from navalia says this:
navalia:chrome :WARN > Retrying 1 time(s) due to issue: 'Error: [object Object]' +10ms
My question is: how do I get visibility into what the error actually is? Is there a way to dump/format the [object Object]?
I have to set some local storage values to chrome and use auth0 or oauth2 authentication, is there a way to have auth support oauth2? or to set local storage values before doing chrome.goto(url)?
This project hasn't been updated in a while. Is it still being maintained? Trying to decide if we should use it for our next project. Thanks!
This issue is to (hopefully) give folks better visibility into the merge status of chromeless. Below is a fairly rough roadmap of what will transpire:
Again, this is a really rough outline of what will happen based upon my current observations at this point in time. It's possible that any one of these points could take longer or happen faster. I'll try and keep folks updated via this issue thread.
As we're transitioning I'll be doing bugfixes and small requests, but the API is pretty locked down at this point in time. Feel free to create an issue in chromeless https://github.com/graphcool/chromeless/issues for any new feature requests. Thanks!
Here is my dream api:
// in jest
describe('app', () =>
it('should not change', async () => {
const myComponentLib = 'localhost:6006'
const filepath = __dirname
// an image?
const actual = await chrome.goto(myComponentLib)
.then(() => chrome.screenshot(filepath))
.then(() => chrome.done());
const expected = await theMostRecentImageOnDisk
expect(actual).toEqual(expected)
// if this throws, prompt me with a comparison of the two images side by side
// let me approve the overwrite or accept the failure
// ala jest snapshots
})
)
i guess the screenshot could be enough if i had a good diffing library, but the hard part is still prompting users with the comparison side-by-side, and then continuing the tests after input is received.
this feels like an overlap of concerns between the test runner, the assertion lib, and the driver, but nobody is doing this well right now. jest html snapshots have the perfect workflow, it's just that they're nonsense text output instead of pictures.
Hello I was trying out Navalia for the first time and it's awesome ๐ I'm just having one problem. Whenever I add chrome.size(1024, 768)
to a test it'll timeout on this part navalia:chrome :screenshot() > saving screenshotpage
I'm using Navalia 1.2.0
and Jest 21.0.1
with Node 8.4.0
.
My test:
import { Chrome } from 'navalia'
import { toMatchImageSnapshot } from 'jest-image-snapshot'
expect.extend({ toMatchImageSnapshot })
jasmine.DEFAULT_TIMEOUT_INTERVAL = 30000
describe('Navalia', () => {
let chrome
beforeEach(() => {
chrome = new Chrome()
})
afterEach(() => {
chrome.done()
})
it('should create a screenshot', async () => {
await chrome.goto('http://localhost:3000')
await chrome.size(1024, 768)
const image = await chrome.screenshot()
expect(image).toMatchImageSnapshot()
})
})
I've tried increasing the timeout interval to 5 minutes but it just stays stuck on the saving screenshot part until it times out. However as soon as I uncomment the chrome.size(1024, 768)
part it'll work without any problems.
I'm not sure if I'm doing anything wrong or if it's an issue with Navalia, any idea what it could be?
There's enough meta-data inside of Chrome to generate a webpage with only the resources it uses. This is fairly easy to achieve with the css
, but gets tricker with js
as the API for interaction with those resources is different (and the file-size of JS can be quite large).
MVP would:
The graphql version referenced by [email protected] and the jest version referenced by [email protected] seem to be incompatible with one another. Gives an error message:
FAIL src/react.spec.js
โ Test suite failed to run
/Users/sgreene/src/tutorials/test-navalia/node_modules/graphql/index.mjs:2
export { graphql, graphqlSync } from './graphql';
^^^^^^
Steps to reproduce:
package.json looks like this:
{
"name": "test-navalia",
"version": "0.1.0",
"private": true,
"dependencies": {
"react": "^16.2.0",
"react-dom": "^16.2.0",
"react-scripts": "1.1.1"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test --env=jsdom",
"eject": "react-scripts eject"
},
"devDependencies": {
"navalia": "^1.3.0"
}
}
Currently, this library seems to rely on chrome-launcher when it comes to connecting to chrome instances. Would it make sense to have alternative way of connecting to already running instances during Navalia initialization? I wouldnt mind working on this because i have huge need for this and i really like navalia's api.
I happen to have use case where i have number of VMs running headless chrome and i can connect to them over the remote debugging protocol. It becomes useful when using chrome from AWS Lambda functions, for example.
Anyway, thanks for the fresh approach in browser driver landscape. I've been looking forward to this for some time by now. Cheers!
Need to implement network requests. API might look like:
tab.onNetworkRequest((src: string) => {});
I think having distinct methods for this type of eventing are more obvious (especially with TS), vs using EventEmitter and emitting events on a particular action.
Is there any API to clear all cookies?
this happens when using setWindowSize. The only way I found to get around this is to insert an artificial waiting time (around 10s) or no to have the windows headeless and resizing by hand... as soon as it moves a pixel it re-renders the webpage and everything gets on the right place
I am not sure why my chrome instance is coming up with a blank window when I run navalia code, but it is. I am simply running the examples provided in this article:
https://codeburst.io/capturing-unused-application-code-2b7594a9fe06
As well, I am also running some code based on the code in that article. Nothing on the screen but tests are passing. Is there anyone else who has seen this problem and can possibly share what they've done to fix it? Thanks in advance!
I've only recently started looking into web scrapers and I've been trying to get some headway with them. Until I stumbled across Navalia I was trying to learn Nightmarejs, but since Navalia can work with multiple tabs I think it's a better scraper for me to cut my teeth on.
What has frustrated me with Nightmare and Navalia is a shortage of simple code examples (looping through pagination links, scraping thumbnail URLs, downloading an image, etc). What tutorials/forum discussions I've found assume a lot of previous knowledge or they're too simple and they just open a page and close it. The programming structure is also very different from what I (and I'd say most people) are used to (asynchronous). So in addition to pouring through the API you also have to get used to the way the programming flow works. This makes for a steep learning curve.
But this all brings me to something that I'd like to contribute to the project: code examples. I'm willing to tinker and toil for weeks if I have to in order to get something to work, so I'm not asking for hand-holding. Once I know how to do a few concrete things I can start to get a sense of how things work and start making my own improvements on them.
I'd like to create code snippets for doing basic tasks on specific websites. casperjs comes with a lot of code examples like this, which helped me a lot. If anyone here is willing to answer basic questions now & then I'm willing to do the work of creating code examples from the perspective of a noob in order to help others get started with Navalia.
Running the following code results into an error because it's accessing a property from an undefined variable:
async function run () {
const browser = new Chrome()
await browser.goto('https://google.com')
const res = await browser.evaluate(() => window.__UNKNOWN_VARIABLE__.x)
console.log(res)
await browser.done()
}
Error:
Error
at Chrome.<anonymous> (D:\...\node_modules\navalia\build\Chrome.js:141:23)
at Generator.next (<anonymous>)
at fulfilled (D:\dev\...\node_modules\navalia\build\Chrome.js:4:58)
at <anonymous>
Is there anyway we could forward chrome's error messages when .evaluate()
ing?
Hello,
Firefox exists in headless mode. I am looking for a library to handle both chrome and firefox. Would it be possible to include both in navalia?
Thank you
When trying to use Navalia, I don't seem to be able to connect to chrome as any operation simply timesout. At first I thought this was an issue with my chrome install, but I can connect to it if I install and use chrome-launcher seperately. I'm using ubuntu 16 with version 60 of google-chrome-stable.
Any ideas what might be going wrong?
Edit: I can get Selenium working with chrome so I don't think it's anything to do with the chrome install.
Hello All, awesome project, so much easier to setup than PhanstomJs + CasperJs.
I am attempting to get a POC of End-to-End testing using Navalia. This is in an effort to modernize our frontend testing process. We are wanting to use Javascript style syntax as is aligns better with our projects.
I get Navalia running in the terminal...
Then I attempt to execute the following basic test:
./test_1.js:
const { Chrome } = require('navalia');
const chrome = new Chrome();
chrome
.goto('https://amazon.com')
.type('input', 'Kindle')
.click('.buy-now')
.end()
.then((responses) => {
console.log(responses); // ['https://www.amazon.com/', true, true, true]
});
...but now what? How do I tell node to execute the ./test_1.js again Navalia and this get a chrome session processing the commands.
Sorry if this is super-noob, my background as a server-side engineer leaves my client side knowledgte lacking.
Thank you in advance for the assistance.
After checking the backlog of issue this is a dependant issue of #72
After updating to Chrome Version 62.0.3202.62 tests started failing.
unexpected server response (404)
at ClientRequest._req.on (node_modules/chrome-remote-interface/node_modules/ws/lib/WebSocket.js:651:26)
at emitOne (events.js:115:13)
at ClientRequest.emit (events.js:210:7)
at HTTPParser.parserOnIncomingClient (_http_client.js:565:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23)
at Socket.socketOnData (_http_client.js:454:20)
at emitOne (events.js:115:13)
at Socket.emit (events.js:210:7)
at addChunk (_stream_readable.js:266:12)
at readableAddChunk (_stream_readable.js:253:11)
Debug log:
navalia starting, using up to 1 instances +0ms
navalia:chrome-helper using up to infinite tabs +1ms
navalia instances are available and starting +2ms
navalia:chrome-helper starting chrome +0ms
navalia instances are available and starting +226ms
ChromeLauncher Waiting for browser. +9ms
ChromeLauncher Waiting for browser... +0ms
ChromeLauncher Waiting for browser..... +512ms
ChromeLauncher Waiting for browser.....โ +2ms
First, awesome project. I have seen notes about this project is or will be deprecated. I do hope rumors of its demise will be proven false. :)
The Chrome
instance is easy enough to use however the navalia
usage seems to have a problem. I haven't dug into code yet though.
This example fails:
process.env.DEBUG = "navalia";
import { Navalia } from "navalia";
const navalia: Navalia = new Navalia();
(async () => {
await navalia.run(async chrome => chrome.goto("http://joelgriffith.net"));
})();
Output:
Wed, 14 Feb 2018 22:01:20 GMT navalia starting, using up to 1 instances
Wed, 14 Feb 2018 22:01:20 GMT navalia instances are available and starting
(node:22949) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: unexpected server response (404)
(node:22949) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.