Giter Site home page Giter Site logo

astral's Introduction

Astral

Astral is a high-level puppeteer/playwright-like library that allows for control over a web browser (primarily for automation and testing). It is written from scratch with Deno in mind.

Usage

Take a screenshot of a website.

// Import Astral
import { launch } from "jsr:@astral/astral";

// Launch the browser
const browser = await launch();

// Open a new page
const page = await browser.newPage("https://deno.land");

// Take a screenshot of the page and save that to disk
const screenshot = await page.screenshot();
Deno.writeFileSync("screenshot.png", screenshot);

// Close the browser
await browser.close();

You can use the evaluate function to run code in the context of the browser.

// Import Astral
import { launch } from "jsr:@astral/astral";

// Launch the browser
const browser = await launch();

// Open a new page
const page = await browser.newPage("https://deno.land");

// Run code in the context of the browser
const value = await page.evaluate(() => {
  return document.body.innerHTML;
});
console.log(value);

// Run code with args
const result = await page.evaluate((x, y) => {
  return `The result of adding ${x}+${y} = ${x + y}`;
}, {
  args: [10, 15],
});
console.log(result);

// Close the browser
await browser.close();

You can navigate to a page and interact with it.

// Import Astral
import { launch } from "jsr:@astral/astral";

// Launch browser in headfull mode
const browser = await launch({ headless: false });

// Open the webpage
const page = await browser.newPage("https://deno.land");

// Click the search button
const button = await page.$("button");
await button!.click();

// Type in the search input
const input = await page.$("#search-input");
await input!.type("pyro", { delay: 1000 });

// Wait for the search results to come back
await page.waitForNetworkIdle({ idleConnections: 0, idleTime: 1000 });

// Click the 'pyro' link
const xLink = await page.$("a.justify-between:nth-child(1)");
await Promise.all([
  page.waitForNavigation(),
  xLink!.click(),
]);

// Click the link to 'pyro.deno.dev'
const dLink = await page.$(
  ".markdown-body > p:nth-child(8) > a:nth-child(1)",
);
await Promise.all([
  page.waitForNavigation(),
  dLink!.click(),
]);

// Close browser
await browser.close();

TODO: Document the locator API.

Advanced Usage

If you already have a browser process running somewhere else or you're using a service that provides remote browsers for automation (such as browserless.io), it is possible to directly connect to its endpoint rather than spawning a new process.

// Import Astral
import { launch } from "jsr:@astral/astral";

// Connect to remote endpoint
const browser = await launch({
  wsEndpoint: "wss://remote-browser-endpoint.example.com",
});

// Do stuff
const page = await browser.newPage("http://example.com");
console.log(await page.evaluate(() => document.title));

// Close connection
await browser.close();

If you'd like to instead re-use a browser that you already launched, astral exposes the WebSocket endpoint through browser.wsEndpoint().

// Spawn a browser process
const browser = await launch();

// Connect to first browser instead
const anotherBrowser = await launch({ wsEndpoint: browser.wsEndpoint() });

astral's People

Contributors

jess182 avatar lino-levan avatar lowlighter avatar marvinhagemeister avatar sigmasd avatar teawinzero avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

astral's Issues

Questions migration from puppeteer to astral

Hi !
I'm looking to possibly use this as a replacement of npm:puppeteer as the later isn't designed for deno which seems to cause some troubles on windows (SIGHUP unsupported, instances not properly closing) and during unit testing (leaking ops)

So I'm considering switching to astral, as the others puppeteer-like libs are either outdated or have some caveats from what I tested.

I have a few questions and possibly features requests (if this is in the project scope) though:

  • How reliable is it for unit testing ? Currently I need to pass { sanitizeResources: false, sanitizeOps: false } with puppeteer but the aim is to be able to have proper testing without any leaking ops (including windows).
  • Would it be possible to avoid the use of Deno.command for the browser installation ? On windows requiring the access of Powershell which creates a big attack vector since you need to give a shell access. I haven't experimented it but it seems that it's possible to use DecompressionStream to unzip the archive natively (ref: https://medium.com/deno-the-complete-reference/zip-and-unzip-files-in-deno-ee282da7369f)
  • Similarly, would it be possible to be able to pass the cache path in getBinary() so it's possible to use the same cache as the app using it and have more controls about deno permissions ?
  • There are some methods that doesn't seems to have any equivalent in astral currently:
  • How do you achieve page.on("console", (message: { text: () => string }) => console.debug("puppeteer: ${message.text()}")) and page.on("pageerror", (error: { message: string }) => console.warn("puppeteer: ${error.message}")) on astral to log console message from the browser to your app ? Is it through page.addEventListener
  • What's the equivalent of omitBackground for screenshot (unless it's enabled by default) ?

Thanks in advance for your answer ๐Ÿ˜„ !

Fail to launch browser without write permission

Scenario

Currently there is a scenario where if the sandbox does not have write permissions it causes an exception when "launching" the browser, as it tries to create a temporary folder for "user data"

image

Questions or feature requests for use in unit testing

Hi there! Thanks for working on this; it's great to see finally a Deno native package for working with a headless browser. Puppeteer has been very frustrating with Deno, and we're at the point now that it's not working at all. Attempting to use npm:[email protected] results type errors due to globals being set by npm Node.js types conflicting with globals set by Deno types. Attempting to use https://deno.land/x/[email protected] results in strange runtime errors now with modern Deno; and it's a huge number of major versions behind the latest Puppeteer version on npm. It seems to be unmaintained.

Consider this issue a question about how Astral can be used to achieve some things I previously had working with Puppeteer (when it was working with earlier Deno versions), and if there is no way, or an inconvenient way, consider it a feature request.

There are 2 main reasons I have been using headless browsers over the years:

  1. Screenshotting web pages, or parts of pages. This is probably straightforward with your current Astral API, so I don't have questions about that right now.
  2. Unit testing modules of a published library to check they function as intended in a browser. You want to be able to functionally create a browser page, do unit tests with it in an async callback, and then after it automatically does cleanup:
    • Code coverage is collected by the browser, and then combined with V8 code coverage data collected by Node.js/Deno when the same modules/functions were run directly in tests, to be able to have an accurate code coverage report of the library after the tests have run.
    • Any console messages emitted in the browser page are forwarded to the Node.js/Deno process and logged via console.group under a message indicating the messages came from the headless browser, using what API (warn, error, log, etc.).
    • Page errors are collected and after the tests callback completes, assert the array of page errors is empty. You don't want to end up in a situation where the unit tests are passing but there are uncaught page errors going on undetected.

How can Astral be used to achieve use-case number 2? Here is how it was achieved using Puppeteer:

https://github.com/jaydenseric/ruck/blob/aa068afa7f0630c4e3e8617b2936210c14815964/test/testPuppeteerPage.mjs

It was very difficult for me to work out at first, but that's just because the Puppeteer API doesn't provide a ergonomic ways to do it. Surely it would not be complicated to provide a specific API for this in Astral; an easy way to create a browser page for use in unit testing.

Questions regarding net navigation

Would it be technically possible to catch any navigation (I assume there's an event for it) and validate it against the Deno.permissions API ? Or is this infeasible ?

I guess it could be tricky to handle the navigation from scripts...

await page.evaluate(() => location.href = "https://example.com")

Also, maybe this is more browser-realated rather than directly with astral, but is it possible to disable some protocol schemes ? (like file:// and data:// for example) ?

`browser.newPage` sometimes fails to detect loaded pages

Opening a new page will occasionally throw errors, with the below output, despite the pages themselves successfully loading. Affected sites include LinkedIn, Indeed, and Glassdoor, and it appears to in some way relate to persistent network requests (to tracking APIs, for instance) on page load.

error: Uncaught (in promise) RetryError: Retrying exceeded the maxAttempts (5).
        throw new RetryError(error, options.maxAttempts);
              ^
    at retry (https://jsr.io/@std/async/0.223.0/retry.ts:143:15)
    at eventLoopTick (ext:core/01_core.js:207:9)
    at async Promise.all (index 0)
    at async Page.goto (https://jsr.io/@astral/astral/0.4.0/src/page.ts:521:5)
    at async Browser.newPage (https://jsr.io/@astral/astral/0.4.0/src/browser.ts:166:7)
    at async file:///home/casval/Dev/other-projects/careerCrawler/escapeHatch/scripts/getIndeedHiringPosts.js:39:14
Caused by: DeadlineError: Deadline
    at https://jsr.io/@std/async/0.223.0/deadline.ts:60:32
    at eventLoopTick (ext:core/01_core.js:207:9)
    at async retry (https://jsr.io/@std/async/0.223.0/retry.ts:140:14)
    at async Promise.all (index 0)
    at async Page.goto (https://jsr.io/@astral/astral/0.4.0/src/page.ts:521:5)
    at async Browser.newPage (https://jsr.io/@astral/astral/0.4.0/src/browser.ts:166:7)
    at async file:///home/casval/Dev/other-projects/careerCrawler/escapeHatch/scripts/getIndeedHiringPosts.js:39:14

This started after I switched to the version on JSR, and looks to be from a call to Celestial.network.enable({})
in browser.ts, introduced in c13b687, just after the last version on /x/. I haven't isolated why, but when I comment this out, it stops happening.

Cannot get Astral to work with `deno test`

Hi there. This seems like a great project.

I'm trying to integrate it into a test suite, but deno test complains that there are operations leaking....

When I run the following code: https://gist.github.com/cowboyd/c5735b5f80caf2e745dd93ee78cc48b7

import { describe, it } from "https://deno.land/[email protected]/testing/bdd.ts"
import { launch } from "https://deno.land/x/[email protected]/mod.ts";

describe("browser", () => {
  it("launches, sleeps, and closes", async () => {
    let browser = await launch();
    await new Promise((resolve) => setTimeout(resolve, 2000));
    browser.close();
  });
});

I consistently see the following:

$ deno test -A browser.test.ts
Check browser.test.ts
running 1 test from browser.test.ts
browser ...
  launches, sleeps, and closes ... FAILED (2s)
browser ... FAILED (due to 1 failed step) (2s)

 ERRORS

browser ... launches, sleeps, and closes => https://deno.land/[email protected]/testing/_test_suite.ts:323:15
error: Leaking async ops:
  - 1 async operation to receive the next message on a WebSocket was started in this test, but never completed. This is often caused by not closing a `WebSocket` or `WebSocketStream`.
  - 1 async operation to op_spawn_wait was started in this test, but never completed.
To get more details where ops were leaked, run again with --trace-ops flag.

Should browser.close() be an async operation that waits until everything has been cleaned up?

Alternative to `await page.setRequestInterception(true);`?

Hello,

on Puppeteer, you can do:

page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (request) => {
    if (['image', 'stylesheet', 'font', 'script'].indexOf(request.resourceType()) !== -1) {
        request.abort();
    } else {
        request.continue();
    }
});

To prevent downloading unwanted assets, how can I do that on Astral?

Astral can't download a supported browser on ARM linux

Hey. I just tried using astral for the first time, but immediately after launching the browser, the script threw an error and crashed.

Code:

import { launch } from "https://deno.land/x/astral/mod.ts";
const browser = await launch();

Error:

Uncaught Error: Failed to spawn '/home/liam/.cache/astral/121.0.6130.0/chrome-linux64/chrome': Exec format error (os error 8)
    at spawnChildInner (ext:runtime/40_process.js:185:17)
    at spawnChild (ext:runtime/40_process.js:206:10)
    at Command.spawn (ext:runtime/40_process.js:474:12)
    at runCommand (https://deno.land/x/[email protected]/src/browser.ts:13:27)
    at launch (https://deno.land/x/[email protected]/src/browser.ts:254:39)
    at eventLoopTick (ext:core/01_core.js:166:7)
    at async <anonymous>:1:38

Environment Information:
Raspberry Pi 4b+ 4GB + 512 GB SSD
Ubuntu Server 23.10
Deno 1.41.1

Export PDF cause `maxAttempts` error

When trying to export a PDF, I encounter a maxAttempts error

// Import Astral
import { launch } from "jsr:@astral/astral";

// Launch the browser
const browser = await launch();

// Open a new page
const page = await browser.newPage("https://deno.land");

// Create a pdf of the page and save that to disk
const pdfPage = await page.pdf();
Deno.writeFileSync("page.pdf", pdfPage);

// Close the browser
await browser.close();

Cause error

error: Uncaught (in promise) RetryError: Retrying exceeded the maxAttempts (5).
        throw new RetryError(error, options.maxAttempts);
              ^

Environnement

astral : v0.4.2
deno : v1.44.1
os : Windows 11, x64

Configurable tempDir or at least exposing it some how

I'm launching several hundred browser sessions, each one utilizes Deno.makeTempDirSync() to get where to store the browsers user-data-dir.
It would be awesome if LaunchOptions had a userDataDir option to override this behavior, so I could have them be generated on a seperate drive that has more disk space available.
At the very least returning or making available which temporary directory it created would be great, so I can properly cleanup the files afterwards.

Evaluating astral for Fresh

I'm currently considering moving to astral for e2e testing for Fresh 2. We currently rely on a very old puppeteer instance. In Fresh we don't need much functionality so this seems like a perfect fit.

Here are some random thoughts while porting the test suite:

  • await(await page.$(".update"))!.click(); works, but a locator API would be a little smoother (that said I don't mind it too much)
  • Not sure how to do history navigation. We navigate forward + backwards in the history stack in some tests
  • Not sure how to capture uncaught exceptions or send something back from the browser to the Deno process. I know how to inject my own scripts into the page, but not sure how to send something back at a later point in time.
  • Page.url can be undefined, but afaik every page in a browser has at least the default url about:blank. Would remove a few null assertions in test cases
  • Would be cool if page.evaluate could infer return type

Overall it's a really nice experience. I love that it auto-installs chromium when not present. That's much better than what we had before. Looking forward to using it more ๐Ÿ‘

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.