experity/imagine

Screenshot webpages and convert to image/pdf

About

"Imagine" is a web service that takes a given url and returns a screenshot in the desired format (e.g. png or pdf).

Setup

First, install Docker. You can find platform specific instructions here.

To build and start up the app in a Docker container run: ./bin/run.sh

You'll now be able to access the app from your browser at $APP_HOST:80.

Tests

Included in the bin directory is a script to run the test suite in Docker: ./bin/test.sh

The test script will rebuild the Docker image before running the suite so the tests should always represent current code. Sometimes npm does not install packages successfully causing a failed build, but this can be dealt with by re-running the Docker build command (or the appropriate script if you're using them).

Usage

Currently there are two routes that are available: /capture and /heartbeat. The heartbeat route is covered in the "Health Check" section below, and the capture route is covered in the following table:

parameter	description	default
type	the file type: png, pdf, jpg, and gif	none
format	response format: file or string (base64 encoded; only for images)	file
resource	base64 encoded url of the page to be screen capped	none
w	viewport width to be passed to phantomjs	1920
h	viewport height to be passed to phantomjs	1080
wait	number of milliseconds to wait after page load before taking screenshot	0
signature	base64 encoded SHA256 digest of resource param + app secret	none

Note: the resource param mentioned above must be base64 encoded to prevent issues with the url.

The signature param is the base64 encoded SHA256 digest of the resource param + the app secret (in that order!) defined in config/app_data.yml. This signature is a way of disallowing service to untrusted traffic. In the event that a request comes in with a bad signature, a 403 status is returned along with the message "Invalid signature."

Examples

Assuming your app secret is secret! and you want to screenshot https://github.com/LightshedHealth/imagine then:

# Example of creating signature in Ruby
require 'base64'
require 'digest'

app_secret = ENV['IMAGINE_APP_SECRET'] # => 'secret!'
url = 'https://github.com/LightshedHealth/imagine'
encoded_url = Base64.strict_encode64(url) # => 'aHR0cHM6Ly9naXRodWIuY29tL0xpZ2h0c2hlZEhlYWx0aC9pbWFnaW5l'
signature = Digest::SHA256.base64digest(encoded_url + app_secret) # => 'AyMK86hre66wous/C5KD2EOYK2+WHk8beSguu4lg0Jg='

To get a png image visit the following url in your browser: $APP_HOST/capture?type=png&resource=aHR0cHM6Ly9naXRodWIuY29tL0xpZ2h0c2hlZEhlYWx0aC9pbWFnaW5l&signature=AyMK86hre66wous/C5KD2EOYK2+WHk8beSguu4lg0Jg=

Troubleshooting

Logs

To inspect the logs of a running container (e.g. a container running the imagine-dev image): docker logs imagine-dev

Interacting with the container

Sometimes it's useful to start an interactive session with your container: docker run --rm -i imagine-dev /bin/bash

In the above example you will likely not see any output until you enter a command (e.g. pwd).

Health Check

If you want to check if the app is up without requesting a screenshot, you can visit $APP_HOST/heartbeat. The "heartbeat" route returns "ok!" with a 200 response code and does not log the request (useful if you've got a monitoring service checking the app and don't want to fill your disk with logs).

imagine's Issues

Root service request should not return error

When requesting just the root URL, you get the following error:

Error: Cannot find module 'ejs'
   at Function.Module._resolveFilename (module.js:339:15)
   at Function.Module._load (module.js:290:25)
   at Module.require (module.js:367:17)
   at require (internal/module.js:16:19)
   at new View (/src/node_modules/express/lib/view.js:78:30)
   at EventEmitter.render (/src/node_modules/express/lib/application.js:569:12)
   at ServerResponse.render (/src/node_modules/express/lib/response.js:961:7)
   at /src/app.js:25:7
   at Layer.handle_error (/src/node_modules/express/lib/router/layer.js:71:5)
   at trim_prefix (/src/node_modules/express/lib/router/index.js:310:13)

It should probably just return a 404.

App crashes after serving up a pdf

When a browser requests a pdf, the app serves up the document and then promptly dies.

Note: this doesn't happen when making the request from the command line (e.g. using curl)

Encode/decode resource param

The resource param can be tricky when the url has query parameters and other valid url attributes. Base 64 encoding the resource param before making the request to the app would go a long way towards ensuring the resource can be retrieved properly.

Arbitrary Code Execution

While looking through and testing, I noticed that there are some exec statements within the main file. Out of curiosity I ran the following command on my test server (host obfuscated for safety reasons)

curl $APP_HOST:80/capture?type=png&format=string&resource=https://github.com;echo 'hello'

What was returned to me was this:

[3] 8411
[4] 8412
hello
[4]   Done                    format=string

We may want to consider some very prejudice white-listing.

Denial of Service

I can get it to spin, spin, spin on a query like:

http://service/capture?type=pdf&resource=http://service/capture%3Ftype=pdf%26resource=http://service

No logs?

Currently the only things that get written to the console are stack traces when the app crashes and the initial "Listening on port..." message.

It would be beneficial to have at least messages on each request that states the route and params that were sent.