schickling/chromeless

🖥 Chrome automation made simple. Runs locally or headless on AWS Lambda.

chromeless's Issues

Document how to actually run the example in the `usage` section

The usage section of the readme shows how to create a file with some code in, how to start chrome headlessly, but not how to actually run the example. Please update the README.

.evaluate should return values

const evalRes = await chromeless
  .goto('https://www.graph.cool')
  .evaluate(() => {
      return new Promise((resolve) => {
          setTimeout(() => {
              resolve('ok');
          }, 10);
      });
  })

console.log(evalRes)

await chromeless.end()

It's possible to return even async stuff as well: https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#method-evaluate. There's a param called awaitPromise to indicate it's async. You can also "assume" it's async by default even if it's not and just return the value immediately.

.size API

Would be nice to have a size method for changing the browser window size (helpful for screenshots and stuff where users want to capture responsive sites)

Can't use screenshot after evaluate

I'm trying to evaluate some code before taking a screenshot, but when I do that the screenshot returns a completely white image.

You can reproduce this by going to https://chromeless.netlify.com and replacing line 5 with:

  .evaluate(() => { console.log('Hello') })

parameterize if new tab should be created

Finish chromeless-playground

Repo: https://github.com/graphcool/chromeless-playground

Demo: https://chromeless-homepage.netlify.com/#src=const%20chromeless%20=%20new%20Chromeless(%7B%20remote:%20true%20%7D)%0A%0Aconst%20screenshot%20=%20await%20chromeless%0A%20%20.goto('https://www.graph.cool')%0A%20%20.scrollTo(0,%202000)%0A%20%20.screenshot()%0A%0Aconsole.log(screenshot)%0A%0Aawait%20chromeless.end()

• Responsiveness
• Adding a new line should respect indentation
• Redo layout
• Auto detect links and make them clickable
• Add download script button
• Better loading spinner

Revisit screenshot handling

Currently we're uploading to our own Graphcool project - maybe we just want to return the b64 string so people can save it with fs.writeFile on their own

Chrome extension to record session + emit code instructions

Similar to https://github.com/segmentio/daydream

Support for authenticated proxies

Does this support authenticated proxies? Similar to how nightmarejs has the .authentication method

Implement non-promise eval.code

Currently it's not a nice experience when you want to evaluate code, but continue with the method chain. This is an example of tests that I implemented with chromeless. Here it gets pretty clear, that we should have a variant of eval.code that returns the Chromeless instance. One possibility would be to call it directly .code()

export default async (cookies: any[]): Promise<any> => {

  const chromeless = new Chromeless({
    runRemote,
    waitTimeout,
    viewport,
  })

  await chromeless
    .cookies.set(cookies)
    .goto(CONSOLE_URL)
    .wait(3200)
    .wait('a[data-test="sidenav-databrowser"]')
    .click('a[data-test="sidenav-databrowser"]')
    .wait(2200)
    .wait('a[data-test="sidenav-databrowser-model-Post"]')
    .click('a[data-test="sidenav-databrowser-model-Post"]')
    .wait(2200)
    .click('div[class^="NewRowInactive__add"]')
    .type('test url')
    .eval.code(() => document.querySelector('div[data-test="new-row-cell-imageUrl"] input').blur())

  await chromeless
    .click('div[data-test="new-row-cell-description"] > div > div')
    .wait(400)
    .wait('div[data-test="new-row-cell-description"] input')
    .type('some description', 'div[data-test="new-row-cell-description"] input')
    .eval.code(() => document.querySelector('div[data-test="new-row-cell-description"] input').blur())

  await chromeless
    .wait(700)
    .click('button[data-test="add-node"]')
    .wait(300)
    .click('div[data-test="edit-field-imageUrl"]')
    .click('div[data-test="cell-imageUrl"]')
    .type('123')
    .eval.code(() => document.querySelector('input').focus())

  return chromeless
    .press(13)
    .wait(400)
    .click('div[data-test="checkbox-row-0"]')
    .wait(400)
    .click('div[data-test="delete-button"]')
    .wait(400)
    .click('div.button.warning')
    .end()
}

If you're interested in more test examples, here are our console tests: https://github.com/graphcool/console/tree/dev/integration-tests/src/tests

Add more examples

Add integration test use case to Examples section in README

Since running integration tests in parallel is one of the great benefits of the project it'd be good to have a small example showing how it can be done. The examples available right now enlighten the uses in automation and scraping, but not so much on tests.

.coverage API

Chrome exposes a mechanism in which to get coverage for scripts. This type of information is crucial to front-end devs as it exposes unused code-paths. Would love to see an API for this

Implement interactive remote REPL mode

Missing Fonts

I tried demo page to get Taiwan google homepage screenshot, but the content have encoding problem.

Form element API's

Things like check, uncheck, select and more for form elements

Implement inject() and refresh()

These are critical for my tests, and could very well be for others. It may not seem so important, but it is when actual pages don't matter. I test some of my libraries with a single instance on about:blank with inject(); refresh()'ing before each test.

https://github.com/stevenvachon/universal-url/blob/master/test.js#L179-L184

show console.log of evaluate

how can i get console.log from .evaluate() ?
i try to write this code, and i got 'undefined'
what's wrong with my code.

const { Chromeless } = require('chromeless');

async function run() {
  const chromeless = new Chromeless()

  const login = await chromeless
    .cookiesClearAll()
    .goto('http://localhost/project/')
    .type('admin, 'input[id="username"]')
    .type('admin', 'input[id="password"]')
    .click('#submit')
    .wait('#last-loaded')
    .click('#last-loaded')
    .wait('#menu-sidebar')
    .evaluate(() => {
      const Links = [].map.call(
        document.querySelector('.main-nav').querySelectorAll('a'),
        a => a.href
      )
      return JSON.stringify(Links)
    })

  console.log(login)

  await chromeless.end()
}

run()
  .catch(console.error.bind(console))

Cleanup Logging

For example introduce debug

Investigate periodic timeouts

Running chromeless remotely seems to timeout periodically.

Scripts that trigger pageloads fail

Not seeing a wait-like API for pageload. Might be nice to have chromeless "pause" execution if a pageload happens or retry failed actions when a pageload occurs

Automatically "wait" for elements to cut down on script noise

If you're writing a test that does a lot of interactive JS, there's going to be a lot of wait-type calls inside of scripts. It might be nice to automatically wait for elements as opposed to having to tell chromeless every time to wait for something.

I would like to see the source code of the demo front end of the website

Is this demo open source? I'd like to look at the code of the page, especially the editor section.

https://chromeless.netlify.com

Make timeout of wait() customizable

I have been playing around with chromeless and I noticed, that there is a timeout for the wait() function (https://github.com/graphcool/chromeless/blob/master/src/api.ts#L21).

Would be great to be able to customize that. Not sure if it's better via an options object directly passed to the wait() function or via a global config object passed to the new Chromeless().

I can also make a PR for the chosen option.

Scroll to click position

When clicking on an element that is not in the screen, chromeless should scroll to that position

Take screenshot of an element

The screenshot() method returns the full-page screenshot, is there any way to get the screenshot of a DIV or any other element?

.visible API

Ensures that's elements are visible (which is a hard thing to implement in a .evaluate as there's a whole host of things that can make something invisible).

ServerlessError: The request signature we calculated does not match the signature you provided.

ServerlessError: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Rewrite README/docs

ECONNREFUSED

Error: connect ECONNREFUSED 127.0.0.1:9222

?

better solution for click

As the Input.dispatchMouseEvent didn't work for all cases, we added the artificial element.click().
This should have a cleaner solution

Save images or links on element

Is there any way to save image or link on an element?
Or maybe I can use evaluate() to download them 🤔

Please add a license

It is unclear what terms this tool is distributed under. Without a license it defaults to being proprietary (All Rights Reserved). Can you please clarify by adding a LICENSE file?

Write tests

Implement missing features

• back
• forward
• refresh
• mousedown
• mouseup
• viewport
• cookies
• pdf
• inject
• mouseover

Auto viewport resize in headless mode

I'm wondering why viewport size is set to 1440x900 by default when in headless mode. Is it not possible to do this in headless mode too?

Replace Ably with AWS IoT

Ideally the entire AWS IoT setup would be contained in the serverless.yml file of the remote serverless stack.

close tabs

when done

viewport is broken. Change documentation or fix documented solution

Currently viewport parameters has to be set as options like this:

https://chromeless.netlify.com/#src=const%20chromeless%20=%20new%20Chromeless(%7B%20remote:%20true,%20viewport:%20%7Bwidth:%201024,%20height:%20800%7D%20%7D)%0A%0Aconst%20screenshot%20=%20await%20chromeless%0A%20%20.goto('https://www.graph.cool')%0A%20%20.scrollTo(0,%202000)%0A%20%20.screenshot

but the documentation states that you should be calling the viewport method:

https://github.com/graphcool/chromeless/blob/master/docs/api.md#viewportwidth-number-height-number

If you do that, the function will time out.

add tests

Refactor constructor arguments

• Remote function name should be injectable and not hardcoded
• Rethink existing options (maybe viewport?)

Allow setting of `headers`

https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-setExtraHTTPHeaders

Consumers should probably be allowed to set headers for requests (helps with functional tests so that you can auth once and use headers on subsequent requests).

HTTPS error on Win7

Hi guys,

Just FYI: the demo playground may not work on some Win7 machines because the Root Cisco Cert you're using to sign your certificate with is not known.

I tried opening the same website on a Virtual Ubuntu on the same machine ­- all works perfectly. Not sure that you want to support all dinosaur-age operating systems, but still letting you know :­-)

More "helper" methods like `attr` and `text`

Evaluate is a good hook for consumers to do what they wish for DOM stuff, but it's probably a good idea to have some sugar methods like attr, text, and html so that folks can write scrapers more easily.

Chromeless docker container

It would be great to quickly spin up a farm of chromeless images to run a test suite in parallel.
Using containers would remove the AWS requirement for scalability.
I'm not a docker expert, just started to evaluate docker-based solution as our test suite has grown significantly in the last couple of months, and we need a more elastic approach.

Use chrome-launcher to launch Chrome when using Chromeless locally

Use chrome-launcher to launch Chrome when using Chromeless locally. Then we don't have to explain to users that they need to have Chrome running for Chromeless to work locally.

Release TODOs

• Finish readme #13
• process.end() on chrome instance. kill chrome each time for now to avoid zombies
• chrome-plugin option to select which functions to wrap

Optional

• refactor the on.message -> filter topic -> message.toString() stuff into a wrapper, like ably with channels, with builtin topic scopes
• optimise what’s included in the deployment package

Investigate virtual time for deterministically replaying the network stack

resources:

• https://groups.google.com/a/chromium.org/forum/?utm_medium=email&utm_source=footer#!topic/headless-dev/v8GtCwr4X_o
• https://docs.google.com/document/d/1y9KDT_ZEzT7pBeY6uzVt1dgKlwc1OB_vY4NZO1zBQmo/edit

How to retrieve html by selector/xpath

?

Garbage urls in `goto` hang

Probably want to fail quickly if goto gets a bad URL (also might want it to return a value for the resolved URL in case of redirects)