ebidel/try-puppeteer

Run Puppeteer code in the cloud

Try Puppeteer!

Run Puppeteer scripts in the cloud.

Develop

Installation:

yarn; yarn install-backend
# or npm i

Backend

The backend is a Docker container which installs the latest Chrome package that works with Puppeteer on Linux.

Note: You'll need to have Docker running before attempting each step in this section.

Building it

yarn build

Running the container

The container can be run in two modes: standalone as an executable, or as a web service.

1. Using the standalone CLI

The first is a "standalone" mode that runs a Puppeteer script from the CLI. It takes a script file as an argument and runs it in the container.

./backend/run_puppeteer.sh your-puppeteer-script.js

2. Running the web service

The second option is running the container as a web server. The endpoint accepts file uploads for running your Puppeteer scripts in the cloud:

Start the server:

cd backend
yarn serve
# yarn restart is handy too. It rebuilds the container and starts the server.

Example - running a Puppeteer script

async function runCode(code) {
  const form = new FormData();
  form.append('file', new Blob([code], {type: 'text/javascript'}));
  const resp = await fetch('http://localhost:8080/run', {method: 'POST', body: form});
  return await resp.json();
}

const code = `
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  console.log(await page.content());
  browser.close();
`;

runCode(code).then(result => {
  if (result.errors) {
    console.error(result.errors);
  }
  console.log(result.log);
});

Notes:

• There's no need to require('puppeteer'). This is done for you on the backend.
• Top-level async/await are supported.

Code editor frontend

Fire up the code editor UI from the main directory:

yarn serve

Then navigate to http://localhost:8081.

Deployment

1. Update the version of Puppeteer used in index.html, include the doc link. TODO: make this automatic.

2. yarn deploy deploys both the frontend and backend services to App Engine Flex. The apps can also be deployed individually:

yarn deploy-frontend
yarn deploy-backend

Notes & Limitations

• By default, Puppeteer launches and uses its own bundled version of Chromium. To use the google-chrome-unstable installed by the container, pass executablePath:

  const browser = await puppeteer.launch({
    executablePath: 'google-chrome-unstable'
  });

try-puppeteer's Issues

Demo page security

Hi!

Thanks for the great demo page. I just built https://github.com/alvarcarto/url-to-pdf-api which uses Puppeteer to produce PDFs. Someone noticed that our API also allows to reveal file contents in the server when setting e.g. file:///etc/hosts as the URL.

You might know this already but I noticed the same is possible with https://try-puppeteer.appspot.com/ demo page. You can reveal files with await page.goto('file:///etc/hosts');

OpenShift

Can I run this code on Openshift?

Missing fonts

I've managed to setup puppeteer with GAE and got it working. However, there are missing fonts - for example where html font family states Georgia, Tahoma, Trebuchet MS, and Verdana these are missing and conversion to PDF will default to Times New Roman. Fonts Times New Roman, Arial, and Courier New all work fine.

Any ideas on how to get the missing fonts installed and working with Puppeteer on GAE?

Thanks.

Broken flexbox layout in Firefox/Edge

STR: Run the default example

AR:

• Result panel stays fixed and doesn't resize to 50%
• Header offset on the right is misaligned.

Chrome Canary | Firefox Nightly

Analysis (thanks to @dholbert):

The overall height of elements in .puppeteer-results make it larger than the viewport height. The layout is optimizing for a Chrome bug that is being fixed: https://bugs.chromium.org/p/chromium/issues/detail?id=596743

Solution:

Adding min-height: 0 to .puppeteer-results fixes the issue.

Editor typeface

Editor typeface is sans-serif and has weird ligatures, while this might be nice for other uses but is not great for writing code.

broken example: screenshot-fullpage.js

at https://try-puppeteer.appspot.com, using example screenshot-fullpage.js

this line fails:

const devices = require('puppeteer/DeviceDescriptors');

error:

Error running your code. ReferenceError: require is not defined

try-puppeteer: (async() => { ... })() fails silently

From @ithinkihaveacat on November 2, 2017 12:33

The following code (abbreviated version of a file from the examples folder) fails silently when entered in the editing component at https://try-puppeteer.appspot.com/:

(async() => {
const browser = await puppeteer.launch();
console.log('Hello, World');
})();

Actual:

No output, no error message.

Expected:

The output Hello, World!.

Copied from original issue: puppeteer/puppeteer#1254

Use dumb-init

https://github.com/Yelp/dumb-init

Fix backend dockerfile

The build script is no longer working.

Some needed changes:

• Update node version on the Dockerfile.
• The command RUN npm install puppeteer will install the puppeteer 3.x.x with can have breaking changes.

---

This is a suggestion for a Dockerfile that worked for me. I changes the node image to use Alpine, and added the "puppeteer": "^2.0.0" dependency on the package.json.

I also changed the following lines on the server.js from:

    code = code.replace(/\.launch\([\w\W]*?\)/g,
        ".launch({args: ['--no-sandbox', '--disable-dev-shm-usage']})");

to:

    code = code.replace(/\.launch\([\w\W]*?\)/g,
        ".launch({executablePath: '/usr/bin/chromium-browser', args: ['--disable-dev-shm-usage']})");

Dockerfile:

FROM node:alpine

# Tell Puppeteer to skip installing Chrome. We'll be using the installed package.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

# Installs latest Chromium (79) package.
RUN apk add --no-cache \
      chromium \
      nss \
      freetype \
      freetype-dev \
      harfbuzz \
      ca-certificates \
      ttf-freefont

ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.1/dumb-init_1.2.1_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init

WORKDIR /usr/src/app

COPY . .
COPY package*.json ./

# Install deps for server.
RUN npm install --only=production

# building the image.
ARG CACHEBUST=1

# Add pptr user.
RUN addgroup -S pptruser && adduser -S -G pptruser pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser ../app

# Run user as non privileged.
USER pptruser

EXPOSE 8080

ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]

Shareable link

Is it possible to generate shareable link from https://try-puppeteer.appspot.com/ ?

It may be useful to show issue reproduction in puppeteer repository. For example in cases like here:
puppeteer/puppeteer#3299 (comment)

Debugging the website locally

First of all, this is a great demo! I am curious what are the steps to debug the website using localhost? Also, do you have to use yarn restart to get any of your coding changes to be applied? It seems like a very slow workflow so I wanted to be sure this was the correct approach. If there is any other documentation on authoring/debugging this, I would greatly appreciate it.

launch() options get nuked

If the user launches with:

const browser = await puppeteer.launch({
    slowMo: 250, // slow down by 250ms
    headless: false,
    ignoreHTTPSErrors: true,
    dumpio: true,
});

We kill all these options adding --no-sandbox. Instead, the options should be captured, parsed, and reset.

How could I build dependency into try-puppeteer docker

Error: Cannot find module 'axios'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at [eval]:2:11
at ContextifyScript.Script.runInThisContext (vm.js:50:33)
at Object.runInThisContext (vm.js:139:38)
at Object. ([eval]-wrapper:6:22)
at Module._compile (module.js:643:30)
at evalScript (bootstrap_node.js:462:27)

Broken example: proxy.js

Looks like proxy.js is broken as a default right now:

Just wanted to report so y'all are aware! 🙏

Using puppeteer.connect(options)

Would it be possible to use this to connect using puppeteer.connect(options)? It would be nice to be able to connect to Chromium instance running in docker.

/dev/shm size

The default 64M dev/shm size is too small for full page shots of large pages - Wa Post, CNN for example.

I'm aware of the following options at the moment but nothing seems turnkey yet:

1. There's talk on chromium channels to potentially take a tmp dir as a cli parameter.
2. GCP could enable increasing the size of dev/shm on docker run. I am assuming they use default size to keep things simple, but IMO it's going to cause an unnecessary and premature exodus of users from app engine onto compute engine.
3. Could modify puppeteer to handle streaming differently.
4. Migrate off app engine custom/docker entirely and onto container engine proper.

Seems like this project could merge with browserless? Browserless is "run anywhere" so could offer a plugin interface to that layer, and "Google App Engine" could be the first implementation of an adapter.

Error running your code. Error: Failed to launch chrome!

Perhaps something got broken?

PSA: Taking down try-puppeteer.appspot.com

Hi folks, we're taking down https://try-puppeteer.appspot.com/ as it's running a quite old version of puppeteer and it's not being maintained.

Of course, this try-puppeteer open source project is still here, which allows anyone to put up their own version. But the public instance will be offline.

Apologies and thanks for understanding.

Run on localhost

How I can install Chrome headless (like “try-puppeteer.appspot.com”) on my localhost to that others clients can access it in my LAN?

Exist some documents that explain this?.

TypeError: Failed to fetch

hi, I installed backend and frontend on my vps, and to visit backend it shows:

"it works!"

which i assume it works, and i run frontend and always got:

TypeError: Failed to fetch

i thought this might be something wrong about my backend, so i set the backend point to try-puppeteer.appspot.com in app.js as below:

const BACKEND_HOST = ( 'https://backend-dot-try-puppeteer.appspot.com');

and still i got the same TypeError: Failed to fetch

any idea what i got wrong?

thanks?

Catch responses

This code return {"log":""}

const browser = await puppeteer.launch();
const page = await browser.newPage();
page.on('response', (response) => {
    console.log(response.url());
    browser.close();
});
await page.goto('some-test-uri', { waitUntil: 'networkidle2' });

So how can I get response urls?

Docker build apt-utils warning

debconf: delaying package configuration, since apt-utils is not installed

dependencies: update puppeteer version

I've already seen #59 (so the public site will be taken down), but it still makes sense to upgrade the Puppeteer version used by try-puppeteer.

Currently, it is using v1.4.0 while v10.4.0 is the latest.

Related to: puppeteer/puppeteer#7639

Example search.js does not work

The search input needs to be updated from '#searchbox input' to '.devsite-searchbox input' then it works

Run without Docker

Is there any straightforward way of running the 'backend service' without docker?

Cheers
Lasse

Potential Memory Leak??

Hi, Thank you for nice puppeteer project.

I finished my puppeteer project by reference backend/server.js file.
In my case, I got a little memory leak. Because I did not call browser.disconnect() before reusing the browser object.

I read the backend/server.js file again. I did not find codes be related with disconnect browser. So I guess maybe try-puppeteer got the same memory leak problem ?

Potential Memory Leak

page.accessibility not working on https://try-puppeteer.appspot.com/

on https://try-puppeteer.appspot.com/

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.google.com');
console.log(!!page.accessibility);
console.log(!!page.coverage);
console.log(!!page.keyboard);
console.log(!!page.mouse);
console.log(!!page.touchscreen);
console.log(!!page.tracing);
await browser.close();

logs:

false
true
true
true
true
true

going by https://pptr.dev/#?product=Puppeteer&version=v1.19.0&show=api-class-page I would think they all should be true.

To verify it is possible, adding this to the above code does get a tree

const snapshot = await page._client.send('Accessibility.getFullAXTree');
console.log(JSON.stringify(snapshot,null,'  '));

Demo page VM can be escaped

Because the demo page uses Node's "vm" package, the context essentially inherits from Object, which inherits from Function, which means we can escape the context and do malicious things (sorry about that):

Running:

new Function(`this.constructor.constructor('return process')().exit()`)

Will kill node.

There's more discussion about that here gf3/sandbox#50

I'd recommend switching it out for "vm2", which is more secure and has a similar API.

try-puppeteer demo site is down?

Hi,
It seems that the live demo website has stopped working.
Steps to reproduce:

1. Navigate to https://try-puppeteer.appspot.com/
2. Hit 'Run It' (to run the screenshot.js example).
3. Fetching takes a lot of time, resulting in: 'TypeError: Failed to fetch'. The request to https://backend-dot-try-puppeteer.appspot.com/run is failed as 502 showing: 502 Bad Gateway
   nginx

Are you award to this issue?
Can this be fixed?

Thanks!

Page content log gets trimmed

Hi,

not sure if you still support the online version of this project, but I just noticed that the "LOG" window always show a trimmed version of the final rendering content. No matter which page loads or its height, the last lines of code are always trimmed out.

Am I right?

Kind regards,

Cannot scroll to the bottom

(Chrome 76 on Windows 10)

none

search.js not working on https://try-puppeteer.appspot.com/

Steps to reproduce

Tell us about your environment:

Puppeteer version: v1.9.0
Platform / OS version: Windows 10/latest chrome/latest firefox as of today
URLs (if applicable): https://try-puppeteer.appspot.com/
Node.js version: nil

What steps will reproduce the problem?

Go to https://try-puppeteer.appspot.com/
Select search.js from examples selection
Hit Run

Please include code that reproduces the issue.

No code needed. Just try search example.

What is the expected result?
Extract the results from the page.

What happens instead?
Error running your code. Error: No node found for selector: #searchbox input

linux_signing_key.pub missing

I'm trying to build the Docker image. The build fails at this step:

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -

It appears that there's nothing at https://dl-ssl.google.com/linux/linux_signing_key.pub – does that location need to be updated in the Dockerfile?

Error: Protocol error (Page.enable): Target closed.

Hey, you have an error when running the basic script on your website, it returns:

Error running your code. Error: Protocol error (Page.enable): Target closed.

Do you know how to fix this?

Hosting services

Hey,

Where does the demo page is hosted? It runs pretty well on large pages.

Thanks

try-puppeteer: require('puppeteer') not supported

From @ithinkihaveacat on November 2, 2017 12:27

https://try-puppeteer.appspot.com/ explains that it's not necessary to require('puppeteer') when pasting code, but it turns out it's not allowed, either. This makes it (slightly!) more difficult to copy and paste code.

Copied from original issue: puppeteer/puppeteer#1253

Failed to Launch Chrome on https://try-puppeteer.appspot.com/

On Canary Build

Google Chrome is up to date
Version 66.0.3344.0 (Official Build) canary (64-bit)

I get same error on Official Build

Google Chrome is up to date
Version 66.0.3344.0 (Official Build) canary (64-bit)

Getting this error in log section:

Try Puppeteer
v1.0.0

google.com
1
const browser = await puppeteer.launch();
2
​
3
const page = await browser.newPage();
4
await page.goto('http://www.google.com');
5
​
6
console.log(await page.content());
7
await page.screenshot({path: 'screenshot.png'});
8
​
9
await browser.close();
10
​
  
 
Error running your code. Error: Failed to launch chrome!
[0210/102207.499068:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499588:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499629:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499653:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499671:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499690:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499709:ERROR:platform_thread_posix.cc(123)] pthread_create: Resource temporarily unavailable (11)
[0210/102207.499726:FATAL:browser_main_loop.cc(1136)] Failed to start the browser thread: id == 6
#0 0x564e021fa2ec base::debug::StackTrace::StackTrace()
#1 0x564e0221173c logging::LogMessage::~LogMessage()
#2 0x564e00f12353 content::BrowserMainLoop::CreateThreads()
#3 0x564e0124a8e7 content::StartupTaskRunner::RunAllTasksNow()
#4 0x564e00f119fc content::BrowserMainLoop::CreateStartupTasks()
#5 0x564e00f162b8 content::BrowserMainRunnerImpl::Initialize()
#6 0x564e060e2059 headless::HeadlessContentMainDelegate::RunProcess()
#7 0x564e01f36f60 content::RunNamedProcessTypeMain()
#8 0x564e01f37975 content::ContentMainRunnerImpl::Run()
#9 0x564e01f40c5f service_manager::Main()
#10 0x564e01f36481 content::ContentMain()
#11 0x564e060e11fa headless::(anonymous namespace)::RunContentMain()
#12 0x564e060e126e headless::HeadlessBrowserMain()
#13 0x564e01f3da6a headless::HeadlessShellMain()
#14 0x564e007c31c5 ChromeMain
#15 0x7f42943a8b45 __libc_start_main
#16 0x564e007c302a _start

Received signal 6
#0 0x564e021fa2ec base::debug::StackTrace::StackTrace()
#1 0x564e021f9e51 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f429a223890 <unknown>
#3 0x7f42943bc067 gsignal
#4 0x7f42943bd448 abort
#5 0x564e021f8aa5 base::debug::BreakDebugger()
#6 0x564e02211b3f logging::LogMessage::~LogMessage()
#7 0x564e00f12353 content::BrowserMainLoop::CreateThreads()
#8 0x564e0124a8e7 content::StartupTaskRunner::RunAllTasksNow()
#9 0x564e00f119fc content::BrowserMainLoop::CreateStartupTasks()
#10 0x564e00f162b8 content::BrowserMainRunnerImpl::Initialize()
#11 0x564e060e2059 headless::HeadlessContentMainDelegate::RunProcess()
#12 0x564e01f36f60 content::RunNamedProcessTypeMain()
#13 0x564e01f37975 content::ContentMainRunnerImpl::Run()
#14 0x564e01f40c5f service_manager::Main()
#15 0x564e01f36481 content::ContentMain()
#16 0x564e060e11fa headless::(anonymous namespace)::RunContentMain()
#17 0x564e060e126e headless::HeadlessBrowserMain()
#18 0x564e01f3da6a headless::HeadlessShellMain()
#19 0x564e007c31c5 ChromeMain
#20 0x7f42943a8b45 __libc_start_main
#21 0x564e007c302a _start
  r8: ffffb302ba74cd30  r9: ffffb302ba74cd20 r10: 0000000000000008 r11: 0000000000000202
 r12: 00007ffe68213798 r13: 00007ffe68213788 r14: 00007ffe68213790 r15: 00007ffe682132f0
  di: 00000000000069c8  si: 00000000000069c8  bp: 00007ffe68213280  bx: 00007ffe682132f0
  dx: 0000000000000006  ax: 0000000000000000  cx: 00007f42943bc067  sp: 00007ffe68213148
  ip: 00007f42943bc067 efl: 0000000000000202 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.


TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

Keep track of launched browsers

Save the websocket connection for clean up later should anything go wrong.

upgrading node and puppeteer

@ebidel any plans to upgrade the node version to node10 and pup to 1.6.0>

Where do I submit security related issues?

Hello,

I'd like to report a potential security vulnerability, but I don't want to make it immediately visible to all.

What is the most appropriate way to submit it privately?

Block-images.js does not work

Logs print Error running your code. Error: SSL Certificate error: ERR_CERT_AUTHORITY_INVALID

Page crashed

I'm having this error while trying to execute the example:

Error running your code. Error: Page crashed!

Error: net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH

Tried below code but getting an error.

const puppeteer = require('puppeteer');
(async () => {
    const browser = await puppeteer.launch({ignoreHTTPSErrors: true, acceptInsecureCerts: true, args: ['--proxy-bypass-list=*', '--disable-gpu', '--disable-dev-shm-usage', '--disable-setuid-sandbox', '--no-first-run', '--no-sandbox', '--no-zygote', '--single-process', '--ignore-certificate-errors', '--ignore-certificate-errors-spki-list', '--enable-features=NetworkService']});
    const page = await browser.newPage();
    try {

        await page.goto('https://www.hostwpsolutions.com/', {waitUntil: 'networkidle2', timeout: 59000});
        const cookies = await page._client.send('Network.getAllCookies');
        JSON.stringify(cookies, null, 4);
    } catch (e) {
        console.log(e);
    }

    await browser.close();
})();

try-puppeteer on Cloud Run v2.0.0

I had fun at the code of try-puppeteer on Cloud Run with the version 2.0.0 of puppeteer

Link : https://frontend-puppeteer-xoqsda64ya-uc.a.run.app/

https://try-puppeteer.appspot.com -> not working ( Error 404 )

your website stoped working
https://try-puppeteer.appspot.com

it shows this:

404. That’s an error.
     The requested URL was not found on this server. That’s all we know.

FR: Add "share snippet" button

It would be awesome to have a "share snippet" button in the try-puppeteer, that would generate a URL for the snippet.

For example, it can embed snippet in the URL itself:

try-puppeteer.appspot.com/?code=require('puppeteer')...

your-puppeteer-script.js

Hey,

thanks for the awesome code, but what does the your-puppeteer-script.js looks like ? I'm failing to follow the start up guide to start up the app.

Can someone help me please ?

Demo "Try Puppeteer" doesn't work

From @ximet on November 10, 2017 16:39

Step to reproduce:

1. Start https://try-puppeteer.appspot.com
2. Click to button "Run It"

Result: LOG "Error running your code. Error: Page crashed!"

Also I change some code, for example change line await page.goto('https://example.com'); to google.com, and also doesn't work

Copied from original issue: puppeteer/puppeteer#1351