egoist/taki

Take a snapshot of any website.

taki

Install

npm i taki

Built on the top of Google's Puppeteer, for a jsdom/chromy version please visit here.

Usage

const { request } = require('taki')

// Prerender this page to static HTML
// Wait for 1s since this page renders remote markdown file
request({ url: 'https://sao.js.org', wait: 1000 }).then((html) => {
  // serialized html string of target url
  console.log(html)
})

NOTE: You need to call cleanup when you no longer use request:

import { cleanup } from 'taki'

// After fetching..
cleanup()

Custom html selector

By default it returns the html for the entire document, but you can specify a selector to get the html for a specific element.

const { request } = require('taki')

request({ url: 'https://example.com', htmlSelector: '.some-element' }).then(
  (html) => {
    console.log(html)
  }
)

Manually take snapshot

By default taki will take a snapshot of the URL when all resources are loaded, if you have control of the website's source code, you can disable that and manually call window.snapshot:

request({
  url: 'http://my-web.com',
  manually: true,
})

And in your website's source code:

fetchSomeData().then(data => {
  this.setState({ data }, () => {
+    window.snapshot && window.snapshot()
  })
})

Alternatively, choose your own method to invoke when your app is ready to return HTML:

request({
  url: 'http://my-web.com',
  manually: 'iamready',
})

Then call window.iamready() instead of window.snapshot() in your app.

Wait

Wait for specific timeout or a CSS selector to appear in dom.

request({
  url,
  // Wait for 3000 ms
  wait: 3000,
  // Or wait for <div class="comments"></div> to appear
  wait: '.comments',
})

This option will be ignored if manually is set.

Minify

Minify HTML.

request({
  url,
  minify: true,
})

Filter resource

We always abort network requests to following types of resource: stylesheet image media font since they're not required to render the page. In addtion, you can use resourceFilter option to abort specfic type of resource:

request({
  url,
  /**
   * @param {Object} context
   * @param {string} context.type - Resource type
   * @see {@link https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#requestresourcetype}
   * @param {string} context.url - Resource URL
   * @see {@link https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#requesturl}
   * @returns {boolean} Whether to load this resource
   */
  resourceFilter({ type, url }) {
    // Return true to load the resource, false otherwise.
  },
})

You can also use blockCrossOrigin: true shortcut to block all cross-origin requests.

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

Author

taki © egoist, Released under the MIT License.
Authored and maintained by egoist with help from contributors (list).

Website · GitHub @egoist · Twitter @_egoistlily

taki's Issues

Use local installed `chromium` if presets

https://www.npmjs.com/package/chromium

Something like

export const getLocalChromiumPath = () => {
  try {
    const { path } = require('chromium');
  	return path;
  } catch(error) {}
};

Would be useful for using in CI env with chromium binary shipped.

Follow URLs

For example:

taki({
  url: `http://localhost:4000`,
  follow: true
})

If <a href="/about"> is found on / , we should crawl that page as well.