Render and parse the DOM from R via phantomjs.
rdom depends on phantomjs, so make sure that is installed and visible to your PATH:
stopifnot(Sys.which("phantomjs") != "")rdom currently isn't on CRAN, but you can install it with devtools
devtools::install_github("cpsievert/rdom")Web scraping packages such as XML, xml2 and rvest allow you to download and parse HTML files, but they lack a browsing engine to fully render the DOM.
To demonstrate the difference, suppose we want to extract the HTML table on this page:
XML::htmlParse() (and rvest::read_html()) returns the HTML page source, which is static, and doesn't contain the <table> element we desire (because JavaScript is modifying the state of the DOM):
XML::htmlParse("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/")<!DOCTYPE html>
<html><body>
A Simple Table made with JavaScript
<p></p>
<script>
function tableCreate(){
var body = document.body,
tbl = document.createElement('table');
for(var i = 0; i < 3; i++){
var tr = tbl.insertRow();
for(var j = 0; j < 3; j++){
var td = tr.insertCell();
td.appendChild(document.createTextNode("Cell"));
}
}
body.appendChild(tbl);
}
tableCreate();
</script>
</body></html>The main function in rdom, rdom(), uses phantomjs to render and return the DOM as an HTML string. Instead of passing the entire DOM as a string from phantomjs to R, you can give rdom() a CSS Selector to extract certain element(s).
tbl <- rdom::rdom("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/", css = "table")
tbl<table>
<tbody>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
</tbody>
</table> In this case, we can use XML::readHTMLTable() (or rvest::html_table()) to convert the <table> node into a data frame.
XML::readHTMLTable(tbl) V1 V2 V3
1 Cell Cell Cell
2 Cell Cell Cell
3 Cell Cell CellAn interesting use case for rdom is for rendering (and testing) shiny apps.
library(shiny)
runExample("01_hello")Listening on http://127.0.0.1:4870Now, in another R session, pass this URL to rdom(). For this example, we'll just return the app's title.
header <- rdom::rdom("http://127.0.0.1:4870", "h2")<h2>Hello Shiny!</h2>This way it's easy to test your shiny apps!
library("testthat")
expect_identical(XML::xmlValue(header), "Hello Shiny!")rdom() is essentially a wrapper around phantomjs' command line interface. So, if you don't need R for your task, it might be more efficient to use the CLI.
$ git clone https://github.com/cpsievert/rdom.git
$ cd rdom
$ phantomjs inst/rdomjs/rdom.js inst/jsTable/jsTable.html table false 1 table.html
$ vi table.html
There are 4 arguments that the rdom.js script will respect:
urla URL of a web page (required).cssa CSS selector.allThis controls whether querySelector or querySelectorAll is used to extract elements from the page.timeoutmaximum time to wait for page to load and render, in seconds.filenameWrite HTML string to a file.
Everytime you call rdom(), it has to initiate a phantomjs process. Also, phantomjs has to open and fully render the site that you provide. If you need to render multiple sites,
Thanks to Winston Chang for webshot which inspired the design of this package.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent