If you'd like to help others in this project, you're more than welcome! I made this project for work and I wanted to make it available for other people, but usually I don't have the time I'd like to have to maintain the project. So, if you're interested, and want to help with this, just let me know :)
It will be mostly having help to answer some doubts and issues.
Thanks!
Version 3.3.0 includes a new feature that allows managing browser tabs.
This new feature comes with some breaking changes that will allow us future scalability.
To avoid problems for people that uses version >= 3.3.0 of this module, we supported those breaking changes with methods that will be deprecated in version 4.0.0.
This is an abstraction to use a Headless version of Google Chrome in a very simple way. I was inspired by the next projects:
- Doffy (https://github.com/qieguo2016/doffy)
- Horseman (https://github.com/johntitus/node-horseman)
- chrome-remote-interface (https://github.com/cyrus-and/chrome-remote-interface)
- lighthouse (https://github.com/googlechrome/lighthouse)
And I had to read a lot here too:
- https://developers.google.com/web/updates/2017/04/headless-chrome
- https://chromedevtools.github.io/devtools-protocol
And you can also use this in heroku thanks to https://github.com/heroku/heroku-buildpack-google-chrome
I built this basically because I got tired of an error I received in an edge case when using PhantomJS (Unhandled reject Error: Failed to load url). So I decided to make my own abstraction, to be used in a heroku app, and simple to use as Horseman.
I didn't have time to document here in the readme, but every method in the source code is documented.
It's really simple to use. I hope I can get some time to make a QuickStart guide + document the API methods here.
You can read my post in Medium about this module: How to tell to a headless Google Chrome to write a post in Medium for you
- Easy to use
- Multiples Tabs navigation
- Private tab navigation (incognito mode)
- Cookies management
- Full-size screenshots
- Export to PDF
- Docker compatible
And comming soon...
- Video capture (yes, you'll be able to record your navigation and save it as a video or gif)
If you want to collaborate with the project, in any way (documentation, examples, fixes, etc), just send a PR :)
If you rock at making tests, it would be very useful if you can help us making this module better. It's not necesary to build all the tests, but if someone knows how to code the base to add tests to this module, it would really help for someone else to start with this part.
Thank you to everyone who already help submitting a PR! :D
Mac: Chrome Headless is shipped in Chrome Canary. You can install it here: https://www.google.com/chrome/browser/canary.html
Linux: Chrome headless is shipped on chrome 59. so you can install Chrome 59 to use the headless mode:
https://askubuntu.com/questions/79280/how-to-install-chrome-browser-properly-via-command-line
sudo apt-get install libxss1 libappindicator1 libindicator7
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome*.deb # Might show "errors", fixed by next line
sudo apt-get install -f
Just add the buildpack for Heroku and vualá! Everything is ready You can check the buildpack repository here: https://github.com/heroku/heroku-buildpack-google-chrome
With the addition of Chrome Remote Interface into Chrome 59, a simple way to install is using the Docker image for Chrome Headless, such as https://hub.docker.com/r/justinribeiro/chrome-headless/ or https://hub.docker.com/r/yukinying/chrome-headless/
If using Docker, in your app, configure for headless as follows:
const browser = new HeadlessChrome({
headless: true,
launchChrome: false,
chrome: {
host: 'localhost',
port: 9222, // Chrome Docker default port
remote: true,
},
browserlog: true
})npm install --save simple-headless-chrome
Thanks to @lewisf, simple-headless-chrome is compatible on NodeJS >= 4! I hope more persons can benefit of this now :)
const HeadlessChrome = require('simple-headless-chrome')
const browser = new HeadlessChrome({
headless: true, // If you turn this off, you can actually see the browser navigate with your instructions,
chrome: {
userDataDir: '/tmp/headlessDataDir' // This can be null, so a tmp folder will be created and then destroyed
}
})Once you have the browser instance, you can call the methods to interact with it.
Injects JavaScript in the page
Modules available: jQuery, jquery, jQuery.slim and jquery.slim
Parameters
moduleOrScriptstring Javascript code, file, url or name of the module to inject.
Examples
inject('jquery')
You can use jsdelivr to inject any npm or github package in the pageinject('https://cdn.jsdelivr.net/npm/lodash@4/lodash.min.js')inject('https://cdn.jsdelivr.net/npm/jquery@3/dist/jquery.min.js')
You can inject a local Javascript fileinject('./custom-file.js')inject(__dirname + '/path/to/file.js')
Note: the path will be resolved with `require.resolve()` so you can include
files that are in `node_modules` simply by installing them with NPMinject('jquery/dist/jquery.min')inject('lodash/dist/lodash.min')Injects a remote script in the page
Parameters
srcstring Url to remote JavaScript file
Examples
injectRemoteScript(https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js)Injects code in the DOM as script tag
Parameters
scriptstring Code to be injected and evaluated in the DOM
Evaluates a fn in the context of the browser
Parameters
fn{function} - The function to evaluate in the browserargs...any {*} - The arguments to pass to the function
Evaluates an async fn in the context of the browser
Parameters
fn{function} - The function to evaluate in the browserargs...any {*} - The arguments to pass to the function
Evaluates a fn in the context of a passed node
Parameters
nodeNodeObject The Node Object used to get the contextfn{function} - The function to evaluate in the browserargs...any {*} - The arguments to pass to the function
Navigates to a URL
Parameters
urlstring The URL to navigate toopt(optional, default{})optionsobject The options object. options:
Properties
timeoutnumber Time in ms that this method has to wait until the "pageLoaded" event is triggered. If the value is 0 or false, it means that it doesn't have to wait after calling the "Page.navigate" method
Get the value of an Node.
Parameters
nodeNodeObject The Node Object
Returns object Object containing type and value of the element
Get the value of an element.
Parameters
selectorstring The target selectorframeIdstring The FrameID where the selector should be searched
Returns object Object containing type and value of the element
Set the value of an element.
Parameters
nodeNodeObject The Node Objectvaluestring The value to set the node to (it may be an array of values when the node is a multiple "HTMLSelectElement")
Set the value of an element.
Parameters
selectorstring The selector to set the value of.valuestring? The value to set the selector toframeIdstring The FrameID where the selector should be searched
Fills a selector of an input or textarea element with the passed value
Parameters
selectorstring The selectorvaluestring The value to fill the element matched in the selectorframeIdstring The FrameID where the selector should be searched
Clear an input field.
Parameters
selectorstring The selector to clear.frameIdstring The FrameID where the selector should be searched
Returns the node associated to the passed selector
Parameters
selectorstring The selector to findframeIdstring The FrameID where the selector should be searched
Focus on an element matching the selector
Parameters
selectorstring The selector to find the elementframeIdstring The FrameID where the selector should be searched
Simulate a keypress on a selector
Parameters
selectorstring The selector to type into.textstring The text to type.frameIdstring The FrameID where the selector should be searchedoptsoptionsobject Lets you send keys like control & shift
Types text (doesn't matter where it is)
Parameters
Select a value in an html select element.
Parameters
selectorstring The identifier for the select element.valuestring The value to select.frameIdstring The FrameID where the selector should be searched
Fire a key event.
Parameters
typestring The type of key event. (optional, defaultkeypress)keystring The key to use for the event. (optional, defaultnull)modifiernumber The keyboard modifier to use. (optional, default0)windowsVirtualKeyCode(optional, default0)
Waits certain amount of ms
Parameters
timenumber Ammount of ms to wait
Binding callback to handle console messages
Parameters
listeneris a callback for handling console message
Waits for a page to finish loading. Throws error after timeout
Parameters
timeoutnumber The timeout in ms. (Default: "loadPageTimeout" property in the browser instance options)
Waits for all the frames in the page to finish loading. Returns the list of frames after that
Parameters
Returns object List of frames, with childFrames
Waits for a selector to finish loading. Throws error after timeout
Parameters
selectorstring The identifier for the select element.intervalnumber The interval in ms. (Default: "loadPageTimeout" property in the browser instance options)timeoutnumber The timeout in ms. (Default: "loadPageTimeout" property in the browser instance options)
Fire a mouse event.
Parameters
$0Object$0.type(optional, default'mousePressed')$0.x(optional, default0)$0.y(optional, default0)$0.modifiers(optional, default0)$0.button(optional, default'left')$0.clickCount(optional, default1)
typestring Type of the mouse event. Allowed values: mousePressed, mouseReleased, mouseMoved. (optional, defaultmousePressed)xnumber X coordinate of the event relative to the main frame's viewport. (optional, default0)ynumber Y coordinate of the event relative to the main frame's viewport. 0 refers to the top of the viewport and Y increases as it proceeds towards the bottom of the viewport. (optional, default0)modifiernumber Bit field representing pressed modifier keys. Alt=1, Ctrl=2, Meta/Command=4, Shift=8 (default: 0). (optional, default0)buttonstring Mouse button (default: "none"). Allowed values: none, left, middle, right. (optional, defaultleft)
Click on a selector by firing a 'click event' directly in the element of the selector
Parameters
selectorstring Selector of the element to clickframeIdstring The FrameID where the selector should be searched
Clicks left button hover the centroid of the element matching the passed selector
Parameters
Calculates the centroid of a node by using the boxModel data of the element
Parameters
nodeIdstring The Node Id
Returns object { x, y } object with the coordinates
Get the browser cookies
Returns object Object with all the cookies
Set the browser cookies
Parameters
namestring The name of the cookie.valuestring The value of the cookie.options(optional, default{})urlstring The request-URI to associate with the setting of the cookie.
Properties
optionsobject Options objectdomainstring? If omitted, the cookie becomes a host-only cookiepathstring? Defaults to the path portion of the url parametersecureboolean? Defaults to false.httpOnlyboolean? Defaults to false.sameSitestring? Represents the cookie's 'SameSite' status: https://tools.ietf.org/html/draft-west-first-party-cookiesexpirationDatenumber? If omitted, cookie becomes a session cookie }} options - additional options for setting the cookie (more info here: https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-setCookie)
Returns boolean True if successfully set cookie
Clear the browser cookies
Checks if an element matches the selector
Parameters
selectorstring The selector stringframeIdstring The FrameID where the selector should be searched
Returns boolean Boolean indicating if element of selector exists or not
Checks if an element matching a selector is visible
Parameters
selectorstring The selector stringframeIdstring The FrameID where the selector should be searched
Returns boolean Boolean indicating if element of selector is visible or not
Takes a screenshot of the page and returns it as a string
Parameters
captureOptionsobject Options object Options properties:captureOptions.format(optional, default'png')captureOptions.qualitycaptureOptions.clip(optional, default{x:0,y:0,width:this.options.deviceMetrics.width,height:this.options.deviceMetrics.height,scale:this.options.deviceMetrics.deviceScaleFactor})captureOptions.fromSurfacecaptureOptions.selectorcaptureOptions.fullPage
returnBinaryboolean If true, returns as binary. Otherwise, returns a base64 string (optional, defaultfalse)
Properties
formatstring? Image compression format (defaults to png). Allowed values: jpeg, png.qualityinteger? Compression quality from range [0..100] (jpeg only).clipViewPort? Capture the screenshot of a given viewport/region only (https://chromedevtools.github.io/devtools-protocol/tot/Page/#type-Viewport)fromSurfaceboolean? Capture the screenshot from the surface, rather than the view. Defaults to false. EXPERIMENTALselectorstring? The selector to be captured. If empty, will capture the pagefullPageboolean? If true, captures the full page height
Returns string Binary or Base64 string with the image data
Saves a screenshot of the page
Parameters
fileNamestring Path and Name of the file (without the extension) (optional, default`screenshot-${Date.now()}`)captureOptionsobject Options object Options properties: (optional, default{})captureOptions.format(optional, default'png')captureOptions.qualitycaptureOptions.clipcaptureOptions.fromSurfacecaptureOptions.selectorcaptureOptions.fullPage
Properties
formatstring Image compression format (defaults to png). Allowed values: jpeg, png.qualityinteger Compression quality from range [0..100] (jpeg only).clipViewPort Capture the screenshot of a given region only (https://chromedevtools.github.io/devtools-protocol/tot/Page/#type-Viewport)fromSurfaceboolean Capture the screenshot from the surface, rather than the view. Defaults to false. EXPERIMENTALselectorstring? The selector to be captured. If empty, will capture the pagefullPageboolean? If true, captures the full page height
Returns string Binary or Base64 string with the image data
Prints the page to PDF
Parameters
options(optional, default{})returnBinaryboolean If true, returns as binary. Otherwise, returns a base64 string (optional, defaultfalse)
Properties
landscapeboolean Paper orientation. Defaults to false.displayHeaderFooterboolean Display header and footer. Defaults to false.printBackgroundboolean Print background graphics. Defaults to false.scalenumber Scale of the webpage rendering. Defaults to 1.paperWidthnumber Paper width in inches. Defaults to 8.5 inches.paperHeightnumber Paper height in inches. Defaults to 11 inches.marginTopnumber Top margin in inches. Defaults to 1cm (~0.4 inches).marginBottomnumber Bottom margin in inches. Defaults to 1cm (~0.4 inches).marginLeftnumber Left margin in inches. Defaults to 1cm (~0.4 inches).marginRightnumber Right margin in inches. Defaults to 1cm (~0.4 inches).pageRangesstring Paper ranges to print, e.g., '1-5, 8, 11-13'. Defaults to the empty string, which means print all pages. }} options - Options object
Returns string Binary or Base64 string with the PDF data
Saves a PDF file of the page
Parameters
fileNamestring Path and Name of the file (optional, default`pdf-${Date.now()}`)options(optional, default{})
Properties
landscapeboolean Paper orientation. Defaults to false.displayHeaderFooterboolean Display header and footer. Defaults to false.printBackgroundboolean Print background graphics. Defaults to false.scalenumber Scale of the webpage rendering. Defaults to 1.paperWidthnumber Paper width in inches. Defaults to 8.5 inches.paperHeightnumber Paper height in inches. Defaults to 11 inches.marginTopnumber Top margin in inches. Defaults to 1cm (~0.4 inches).marginBottomnumber Bottom margin in inches. Defaults to 1cm (~0.4 inches).marginLeftnumber Left margin in inches. Defaults to 1cm (~0.4 inches).marginRightnumber Right margin in inches. Defaults to 1cm (~0.4 inches).pageRangesstring Paper ranges to print, e.g., '1-5, 8, 11-13'. Defaults to the empty string, which means print all pages. }} options - PDF options
Get the Viewport of the element matching a selector
Parameters
selectorstring The selector stringframeIdstring The FrameID where the selector should be searched
Returns Viewport Object with the viewport properties (https://chromedevtools.github.io/devtools-protocol/tot/Page/#type-Viewport)
Get the list of frames in the loaded page
Returns object List of frames, with childFrames
Resize viewports of the page to full screen size
Accepts or dismisses a JavaScript initiated dialog (alert, confirm, prompt, or onbeforeunload)
Parameters
acceptboolean Whether to accept or dismiss the dialog (optional, defaulttrue)promptTextstring? The text to enter into the dialog prompt before accepting. Used only if this is a prompt dialog. (optional, default'')
Post data from the browser context
Parameters
urlstring The URL or path to POST todataobject? The data object to be posted (optional, default{})optionsobject? Options of the request (optional, default{})
Returns object Request status and data
TODO: Take the value from the DOM Node. For some reason, there're some pages where is not possible to get the textarea value, as its nodeId refreshes all the time
TODO: Take the value from the DOM Node. For some reason, there're some pages where is not possible to get the textarea value, as its nodeId refreshes all the time
Checks if the browser is initialized. Exits the process if it's not
As the selectors may contain colons, it's necessary to escape them in order to correctly match an element
Parameters
selectorstring The selector string
Returns string The selector with colons escaped (One backslash to escape the ':' for CSS, and other to escape the first one for JS)
Runs a promise and throws an error if it's not resolved before the timeout
Parameters
Transforms an interleave array into a key - value object
Parameters
interleaveArrayarray The interleave array
Returns object The key value object
Given an object, transforms it's properties to a URL encoded string
Parameters
objectobject The object to transform
Returns string The URL Enconded object
Creates some delay
Parameters
delaynumber Delay in miliseconds
Returns promise The promise that will solve after the delay
const HeadlessChrome = require('simple-headless-chrome')
const browser = new HeadlessChrome({
headless: true // If you turn this off, you can actually see the browser navigate with your instructions
// see above if using remote interface
})
async function navigateWebsite() {
try {
await browser.init()
const mainTab = await browser.newTab({ privateTab: false })
// Navigate to a URL
await mainTab.goTo('http://www.mywebsite.com/login')
// Fill an element
await mainTab.fill('#username', 'myUser')
// Type in an element
await mainTab.type('#password', 'Yey!ImAPassword!')
// Click on a button
await mainTab.click('#Login')
// Log some info in your console
await mainTab.log('Click login')
// Wait some time! (2s)
await mainTab.wait(2000)
// Log some info in your console, ONLY if you started the app in DEBUG mode (DEBUG='HeadlessChrome*' npm start)
await mainTab.debugLog('Waiting 5 seconds to give some time to all the redirects')
// Navigate a little...
await mainTab.goTo('http://www.mywebsite.com/myProfile')
// Check the select current value
const myCurrentSubscriptionPlan = await mainTab.getValue('#subscriptionSelect')
console.log(myCurrentSubscriptionPlan) // {type: 'string', value: '1 month' }
// Edit the subscription
await mainTab.select('#subscriptionSelect', '3 months')
await mainTab.click('#Save')
// Resize the viewport to full screen size (One use is to take full size screen shots)
await mainTab.resizeFullScreen()
// Take a screenshot
await mainTab.saveScreenshot('./shc.png')
// Get a HTML tag value based on class id
const htmlTag = await mainTab.evaluate(function(selector) {
const selectorHtml = document.querySelector(selector)
return selectorHtml.innerHTML
}, '.main'); // returns innerHTML of first matching selector for class "main"
// Close the browser
await browser.close()
} catch (err) {
console.log('ERROR!', err)
}
}
navigateWebsite()- .waitForSelector
- .setCookie (set individual cookie) Thanks @saidganim !
- .setCookies (set a full object of cookies, like the one from .getCookies())
- --disable-translate
- --disable-extensions
- --no-first-run
- And many more! Only those useful... All supported thanks to @hugorodrigues. Now just pass an array in the init settings, like this:
const browser = new HeadlessChrome({
headless: false, // If you turn this off, you can actually see the browser navigate with your instructions
chrome: {
flags: [
'--use-fake-device-for-media-stream',
'--use-fake-ui-for-media-stream'
]
}
})-
Handle xpath besides regular selectors
-
Separate the methods in the actions file in actions per Domain (see left menu here: https://chromedevtools.github.io/devtools-protocol/tot/)
-
Allow adding new targets/tabs and controlling them at the same time (https://github.com/cyrus-and/chrome-remote-interface#cdpnewoptions-callback and https://github.com/cyrus-and/chrome-remote-interface/wiki/Inspect-a-new-tab). Thanks @iyttor ! This was a great contribution! :D
-
Improve existing methods: .getCookies - Should receive a cookie name and return only that one, or all the cookies if no key is specified
-
Bypass Certificate Errors (https://github.com/cyrus-and/chrome-remote-interface/wiki/Bypass-certificate-errors-(%22Your-connection-is-not-private%22)) Thanks @trevan !
-
Add Target domain API So we can create tabs: https://chromedevtools.github.io/devtools-protocol/tot/Target/#method-createTarget
I was thinking on using this HTML page to make all the tests: https://github.com/cbracco/html5-test-page
It'd be great to have some unit tests for each HTML element; besides, those test may be useful examples for everyone.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent