lightpanda-io / browser Goto Github PK

View Code? Open in Web Editor NEW

78.0 3.0 0.0 1.64 MB

The open-source browser made for headless usage

Home Page: https://lightpanda.io

License: GNU Affero General Public License v3.0

Zig 98.99% Makefile 0.91% HTML 0.10%

browser cdp headless playwright puppeteer zig

browser's Introduction

Lightpanda

Lightpanda is the open-source browser made for headless usage:

Javascript execution
Support of the Web APIs (partial, WIP)
Compatible with Playwright, Puppeteer through CDP (WIP)

Fast scraping and web automation with minimal memory footprint:

Ultra-low memory footprint (12x less than Chrome)
Blazingly fast & instant startup (64x faster than Chrome)

See benchmark details.

Why?

Javascript execution is mandatory for the modern web

Back in the good old times, grabbing a webpage was as easy as making an HTTP request, cURL-like. It’s not possible anymore, because Javascript is everywhere, like it or not:

Ajax, Single Page App, Infinite loading, “click to display”, instant search, etc.
JS web frameworks: React, Vue, Angular & others

Chrome is not the right tool

So if we need Javascript, why not use a real web browser. Let’s take a huge desktop application, hack it, and run it on the server, right? Hundreds of instance of Chrome if you use it at scale. Are you sure it’s such a good idea?

Heavy on RAM and CPU, expensive to run
Hard to package, deploy and maintain at scale
Bloated, lots of features are not useful in headless usage

Lightpanda is built for performance

If we want both Javascript and performance, for a real headless browser, we need to start from scratch. Not yet another iteration of Chromium, really from a blank page. Crazy right? But that’s we did:

Not based on Chromium, Blink or WebKit
Low-level system programming language (Zig) with optimisations in mind
Opinionated, no rendering

Status

Lightpanda is still a work in progress and is currently at the Alpha stage.

Here are the key features we want to implement before releasing a Beta version:

We will not provide binary versions until we reach at least the Beta stage.

NOTE: There are hundreds of Web APIs. Developing a browser, even just for headless mode, is a huge task. It's more about coverage than a working/not working binary situation.

You can also follow the progress of our Javascript support in our dedicated zig-js-runtime project.

Build from sources

We do not provide yet binary versions of Lightpanda, you have to compile it from source.

Prerequisites

Lightpanda is written with Zig 0.13.0. You have to install it with the right version in order to build the project.

Lightpanda also depends on zig-js-runtime (with v8), Netsurf libs and Mimalloc.

To be able to build the v8 engine for zig-js-runtime, you have to install some libs:

For Debian/Ubuntu based Linux:

sudo apt install xz-utils \
    python3 ca-certificates git \
    pkg-config libglib2.0-dev \
    gperf libexpat1-dev \
    cmake clang

For MacOS, you only need cmake:

brew install cmake

Install and build dependencies

All in one build

You can run make install to install deps all in one (or make install-dev if you need the development versions).

Be aware that the build task is very long and cpu consuming, as you will build from sources all dependancies, including the v8 Javascript engine.

Step by step build dependancy

The project uses git submodules for dependencies.

To init or update the submodules in the vendor/ directory:

make install-submodule

Netsurf libs

Netsurf libs are used for HTML parsing and DOM tree generation.

make install-netsurf

For dev env, use make install-netsurf-dev.

Mimalloc

Mimalloc is used as a C memory allocator.

make install-mimalloc

For dev env, use make install-mimalloc-dev.

Note: when Mimalloc is built in dev mode, you can dump memory stats with the env var MIMALLOC_SHOW_STATS=1. See https://microsoft.github.io/mimalloc/environment.html.

zig-js-runtime

Our own Zig/Javascript runtime, which includes the v8 Javascript engine.

This build task is very long and cpu consuming, as you will build v8 from sources.

make install-zig-js-runtime

For dev env, use make iinstall-zig-js-runtime-dev.

Test

Unit Tests

You can test Lightpanda by running make test.

Web Platform Tests

Lightpanda is tested against the standardized Web Platform Tests.

The relevant tests cases are committed in a dedicated repository which is fetched by the make install-submodule command.

All the tests cases executed are located in the tests/wpt sub-directory.

For reference, you can easily execute a WPT test case with your browser via wpt.live.

Run WPT test suite

To run all the tests:

make wpt

Or one specific test:

make wpt Node-childNodes.html

Add a new WPT test case

We add new relevant tests cases files when we implemented changes in Lightpanda.

To add a new test, copy the file you want from the WPT repo into the tests/wpt directory.

⚠️ Please keep the original directory tree structure of tests/wpt.

browser's People

Contributors

Stargazers

Watchers

browser's Issues

DOM Events

node.textContent with an null parent generates a crash

The following JS code generates a crash (this code is present in tests/wpt/dom/nodes/Node-textContent.html test cases)

  var text = document.createTextNode("abc");
  text.textContent = "def";

$ make wpt tests/wpt/dom/nodes/Node-textContent.html                                                                                                                                                                                                                             [33/9180]
Building wpt...                                                       
steps [8/11] install browsercore-wpt... Segmentation fault at address 0x0
/home/pierre/wrk/browsercore/vendor/netsurf/libdom/include/dom/events/event_target.h:75:40: 0xa99767 in dom_event_target_dispatch_event (src/events/dispatch.c)
return ((dom_event_target_vtable *) et->vtable)->dispatch_event(                                                                            
                                      ^                                                                                                     
src/events/dispatch.c:224:8: 0xa99bcf in __dom_dispatch_subtree_modified_event (src/events/dispatch.c)                                       
src/core/characterdata.c:151:9: 0xa9afe1 in _dom_characterdata_set_data (src/core/characterdata.c)
/home/pierre/wrk/browsercore/vendor/netsurf/libdom/include/dom/core/characterdata.h:60:10: 0xa9ae23 in dom_characterdata_set_data (src/core/characterdata.c)
return ((dom_characterdata_vtable *) ((dom_node *) cdata)->vtable)-> 
        ^                                                                                                                                   
src/core/characterdata.c:474:9: 0xa9b615 in _dom_characterdata_set_text_content (src/core/characterdata.c)                                   
/home/pierre/wrk/browsercore/src/netsurf.zig:618:61: 0xc90f2e in nodeSetTextContent (browsercore-wpt)
   const err = nodeVtable(node).dom_node_set_text_content.?(node, s);                                                                       
                                                           ^                                                                                
/home/pierre/wrk/browsercore/src/dom/node.zig:159:45: 0xc90ff9 in set_textContent (browsercore-wpt)                                          
       return try parser.nodeSetTextContent(self, data);                                                                                    
                                           ^                                                                                                
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/generate.zig:704:9: 0xbd2178 in callFunc__anon_123799 (browsercore-wpt)     
       res = @call(.auto, function, args) catch |err| {                                                                                     
       ^                                                                                                                                    
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/generate.zig:838:21: 0xba3bf1 in setter (browsercore-wpt)                   
           callFunc(                                                                                                                        
                   ^                                                                                                                        
../../../../v8/src/api/api-arguments-inl.h:332:3: 0x19b661d in CallAccessorSetter (../../../../v8/src/objects/objects.cc)  
../../../../v8/src/objects/objects.cc:1521:34: 0x19b61ec in SetPropertyWithAccessor (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:2560:16: 0x19bd232 in SetPropertyInternal (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:2635:9: 0x19bcc27 in SetProperty (../../../../v8/src/objects/objects.cc)
../../../../v8/src/ic/ic.cc:1851:5: 0x2a2cfc6 in Store (../../../../v8/src/ic/ic.cc)                         
../../../../v8/src/ic/ic.cc:2802:3: 0x2a37741 in __RT_impl_Runtime_StoreIC_Miss (../../../../v8/src/ic/ic.cc)
../../../../v8/src/ic/ic.cc:2775:1: 0x2a36db7 in Runtime_StoreIC_Miss (../../../../v8/src/ic/ic.cc)                                          
???:?:?: 0x12ed5fe in ??? (???)                                                                                                              
Unwind error at address `:0x12ed5fe` (error.AddressOutOfRange), trace may be incomplete                                                      
                                                                     
???:?:?: 0x1725c00 in ??? (???)
???:?:?: 0xf56e8d in ??? (???)                                        
???:?:?: 0x7f27a0090d3a in ??? (???)
???:?:?: 0x7f27a009ad7c in ??? (???)
???:?:?: 0xf56e8d in ??? (???)                                        
???:?:?: 0xf4f69b in ??? (???)                                        
???:?:?: 0xf4f3c6 in ??? (???)                                        
../../../../v8/src/execution/simulator.h:155:12: 0x1d5dcfa in Invoke (../../../../v8/src/execution/execution.cc)
../../../../v8/src/execution/execution.cc:538:10: 0x1d5ed7c in CallScript (../../../../v8/src/execution/execution.cc)
../../../../v8/src/api/api.cc:2272:7: 0xe9c9ad in Run (../../../../v8/src/api/api.cc)
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/binding.cpp:414:54: 0xdd90b3 in v8__Script__Run (/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/binding.cpp)
   return maybe_local_to_ptr(ptr_to_local(&script)->Run(ptr_to_local(&context)));
                                                    ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/v8.zig:1687:30: 0xb30ceb in run (browsercore-wpt)
       if (c.v8__Script__Run(self.handle, ctx.handle)) |value| {
                            ^                                        
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:471:28: 0xb31845 in exec (browsercore-wpt)
       const res = scr.run(js_ctx) catch {
                          ^                                          
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:335:21: 0xaee37e in run (browsercore-wpt)
       try res.exec(alloc, script, name, self.isolate, self.js_ctx.?, try_catch);
                   ^                                                 
/home/pierre/wrk/browsercore/src/wpt/run.zig:136:16: 0xaee088 in evalJS (browsercore-wpt)
   try env.run(alloc, script, name, &res, null);
              ^                                                      
/home/pierre/wrk/browsercore/src/wpt/run.zig:99:25: 0xaf2db1 in run__anon_117731 (browsercore-wpt)
       res = try evalJS(js_env, alloc, src, "");
                       ^                                             
/home/pierre/wrk/browsercore/src/main_wpt.zig:126:28: 0xafc3f2 in main (browsercore-wpt)
       const res = wpt.run(&arena, apis, wpt_dir, tc, &loader) catch |err| {
                          ^                                          
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/std/start.zig:585:37: 0xafff86 in main (browsercore-wpt)
           const result = root.main() catch |err| {
                                   ^

make: Nothing to be done for 'tests/wpt/dom/nodes/Node-textContent.html'.

netsurf: create an internal alias `NodeExternal` to avoid using c pointer

We already talked about creating an internal alias NodeExternal to avoid using c pointer. Let's do that in a separate PR.

I think once we have the NodeExternal type we can remove those comments who are not so useful.

Originally posted by @francisbouvier in #100 (comment)

libdom: unref nodes and strings

netsurf/libdom uses a ref counter to track node and string usage an cleanup memory once the ref count reach 0.

see:

We have to ensure our usages of strings and nodes unref correctly once they are done.

To help detect string leak, we can USS a leak detector the same way netsurf does in its example: https://github.com/Browsercore/libdom/blob/master/examples/dom-structure-dump.c#L372

Document properties and methods

https://dom.spec.whatwg.org/#document

Override parser malloc/free

lexbor allow to redefine memory management functions, see https://lexbor.com/docs/lexbor/#memory and the lexbor_memory_setup function.

We could use this to provide our Zig allocator (ie. the Arena allocator) and unify all allocation mechanism from the parser and the engine.

How to handle JS constructors of the DOM API?

For now we consider that the DOM API are created either:

at the parsing stage
through an existing DOM API method (eg. document.createElement)

But some DOM API can be instantiated in JS without an existing node, as standalone node for example let text = new Text().

How can we handle those?
libdom does not provide any function to create a node from scratch.

Should we just allocate the pointer in the heap? On the zig code (easier) or in the C code to use the same allocation mechanism than other nodes?

If so what's going on when we try to attach those standalone nodes in the existing DOM API, for example body.appendChild(text)?

Covered by #102

parser.nodeTextContent returns a partial result

In some cases, maybe due to a text length, the function parser.nodeTextContent returns the text partially.

Example with the HTML https://wpt.live/dom/nodes/ChildNode-after.html when I try to get the script tag text content, I get a partial text.

const scripts = parser.documentGetElementsByTagName(doc, "script");
const slen = parser.nodeListLength(scripts);
for (0..slen) |i| {
    const s = parser.nodeListItem(scripts, @intCast(i)).?;
    const text = parser.nodeFirstChild(s) orelse continue;
    const src = parser.nodeTextContent(text).?;
    std.debug.print("----\n{s}\n-----\n", .{src});
}

The result is:

----


function test_after(child, nodeName, innerHTML) {

    test(function() {
        var parent = document.createElement('div');
        parent.appendChild(child);
        child.after();
        assert_equals(parent.innerHTML, innerHTML);
    }, nodeName + '.after() without any argument.');

    test(function() {
        var parent = document.createElement('div');
        parent.appendChild(child);
        child.after(null);
        var expected = innerHTML + 'null';
        assert_equals(parent.innerHTML, expected);
    }, nodeName + '.after() with null as an argument.');

    test(function() {
        var parent = document.createElement('div');
        parent.appendChild(child);
        child.after(undefined);
        var expected = innerHTML + 'undefined';
        assert_equals(parent.innerHTML, expected);
    }, nodeName + '.after() with undefined as an argument.');

    test(function() {
        var parent = document.createElement('div');
        parent.appendChild(child);
        child.after('');
        assert_equals(parent.lastChild.data, '');
    }, nodeName + '.after() with the empty string as an argument.');

    test(function() {
        var parent = document.createElement('div');
        parent.appendChild(child);
        child.after('text');
        var expected = innerHTML + 'text';
        assert_equals(parent.innerHTML, expected);
    }, nodeName + '.after() with only text as an argument.');

    test(function() {
        var parent = document.createElement('div');
        var x = document.createElement('x');
        parent.appendChild(child);
        child.after(x);
        var expected = innerHTML + '
-----

Check and return error when calling netsurf functions

Attr properties and methods

HTMLDocument properties and methods

https://html.spec.whatwg.org/#the-document-object

dom: processus instruction cloneNode crash

diff --git a/src/dom/document.zig b/src/dom/document.zig
index c99cc5f..811df9d 100644
--- a/src/dom/document.zig
+++ b/src/dom/document.zig
@@ -348,6 +348,7 @@ pub fn testExecFn(
     var createProcessingInstruction = [_]Case{
         .{ .src = "let pi = document.createProcessingInstruction('foo', 'bar')", .ex = "undefined" },
         .{ .src = "pi.target", .ex = "foo" },
+        .{ .src = "let pi2 = pi.cloneNode()", .ex = "undefined" },
     };
     try checkCases(js_env, &createProcessingInstruction);
 
diff --git a/src/dom/processing_instruction.zig b/src/dom/processing_instruction.zig
index 2fa24ee..6a5e6f6 100644
--- a/src/dom/processing_instruction.zig
+++ b/src/dom/processing_instruction.zig
@@ -1,6 +1,7 @@
 const std = @import("std");
 
 const parser = @import("../netsurf.zig");
+const CharacterData = @import("character_data.zig").CharacterData;
 
 // https://dom.spec.whatwg.org/#processinginstruction
 pub const ProcessingInstruction = struct {
@@ -14,7 +15,7 @@ pub const ProcessingInstruction = struct {
     // In consequence, for now, we don't implement all these func for
     // ProcessingInstruction.
     //
-    //pub const prototype = *CharacterData;
+    pub const prototype = *CharacterData;
 
     pub const mem_guarantied = true;

$ make test
Testing...
run test: error:
Generate Union: OK
Generate Tuple: OK
Segmentation fault at address 0x7fff82f0f060
???:?:?: 0x7fff82f0f060 in ??? (???)
Unwind information for `???:0x7fff82f0f060` was not available, trace may be incomplete

/home/pierre/wrk/browsercore/src/dom/node.zig:50:43: 0xbb7a26 in toInterface (test)
        return switch (try parser.nodeType(node)) {
                                          ^
/home/pierre/wrk/browsercore/src/dom/node.zig:172:36: 0xbe9b95 in _cloneNode (test)
        return try Node.toInterface(clone);
                                   ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/generate.zig:781:13: 0xb68e9e in method (test)
            const res = @call(.auto, method_func, args);
            ^
../../../../v8/src/api/api-arguments-inl.h:146:3: 0x26e9039 in Call (../../../../v8/src/builtins/builtins-api.cc)
../../../../v8/src/builtins/builtins-api.cc:113:36: 0x26e79ef in HandleApiCallHelper<false> (../../../../v8/src/builtins/builtins-api.cc)
../../../../v8/src/builtins/builtins-api.cc:148:5: 0x26e6150 in Builtin_Impl_HandleApiCall (../../../../v8/src/builtins/builtins-api.cc)
../../../../v8/src/builtins/builtins-api.cc:135:1: 0x26e5c16 in Builtin_HandleApiCall (../../../../v8/src/builtins/builtins-api.cc)
???:?:?: 0x11b94be in ??? (???)
???:?:?: 0xe22e8d in ??? (???)
???:?:?: 0xe1b69b in ??? (???)
???:?:?: 0xe1b3c6 in ??? (???)
../../../../v8/src/execution/simulator.h:155:12: 0x1c29cfa in Invoke (../../../../v8/src/execution/execution.cc)
../../../../v8/src/execution/execution.cc:538:10: 0x1c2ad7c in CallScript (../../../../v8/src/execution/execution.cc)
../../../../v8/src/api/api.cc:2272:7: 0xd68bed in Run (../../../../v8/src/api/api.cc)
/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp:414:54: 0xca5423 in v8__Script__Run (/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp)
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/v8.zig:1679:30: 0xafb75b in run (test)
        if (c.v8__Script__Run(self.handle, ctx.handle)) |value| {
                             ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:449:28: 0xafc2b5 in exec (test)
        const res = scr.run(context) catch {
                           ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:328:21: 0xac0a87 in run (test)
        try res.exec(alloc, script, name, self.isolate, self.context.?, try_catch);
                    ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/tests/test_utils.zig:67:23: 0xac0417 in checkCases (test)
        try js_env.run(fba_alloc, case.src, name, &res, &cbk_res);
                      ^
/home/pierre/wrk/browsercore/src/dom/document.zig:289:19: 0xac1adc in testExecFn__anon_97462 (test)
    try checkCases(js_env, &createProcessingInstruction);
                  ^
/home/pierre/wrk/browsercore/src/run_tests.zig:47:15: 0xac311a in testExecFn__anon_97444 (test)
    try execFn(alloc, js_env, apis);
              ^
/home/pierre/wrk/browsercore/src/run_tests.zig:69:23: 0xad069c in testsAllExecFn__anon_97442 (test)
        try testExecFn(alloc, js_env, apis, testFn);
                      ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engine.zig:47:18: 0xad0dac in loadEnv__anon_97274 (test)
    try ctxExecFn(alloc, &js_env, apis);
                 ^
/home/pierre/wrk/browsercore/src/run_tests.zig:90:26: 0xad20e0 in test_0 (test)
    try jsruntime.loadEnv(&arena_alloc, testsAllExecFn, apis);
                         ^
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/test_runner.zig:101:29: 0xb056d4 in mainServer (test)
                test_fn.func() catch |err| switch (err) {
                            ^
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/test_runner.zig:34:26: 0xad2e5e in main (test)
        return mainServer() catch @panic("internal test runner failure");
                         ^
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/std/start.zig:575:22: 0xad2964 in main (test)
            root.main();
                     ^
run test: error: while executing test 'test_0', the following command terminated with signal 6 (expected exited with code 0):
/home/pierre/wrk/browsercore/zig-cache/o/f87da788c5a369e80c4398bdaf8ba6c0/test --listen=-
Build Summary: 2/4 steps succeeded; 1 failed; 1/1 tests passed (disable with --summary none)
test transitive failure
└─ run test failure
error: the following build command failed with exit code 1:
/home/pierre/wrk/browsercore/zig-cache/o/098e1df36c28355b4454108f4c97bef1/build /usr/local/zig-0.12.0-dev.1773+8a8fd47d2/zig /home/pierre/wrk/browsercore /home/pierre/wrk/browsercore/zig-cache /home/pierre/.cache/zig --seed 0x49e15c3 test -Dengine=v8
Test ERROR
make: *** [Makefile:86: test] Error 1

dom: text: get_wholeText crash

$ make wpt --  tests/wpt/dom/nodes/Node-cloneNode.html
Building wpt...
steps [8/11] install browsercore-wpt... Segmentation fault at address 0x28
src/core/text.c:480:12: 0xa7bbab in walk_logic_adjacent_text (src/core/text.c)
src/core/text.c:257:9: 0xa7b73a in _dom_text_get_whole_text (src/core/text.c)
/home/pierre/wrk/browsercore/src/netsurf.zig:806:59: 0xc6c220 in textWholdeText (browsercore-wpt)
    const err = textVtable(text).dom_text_get_whole_text.?(text, &s);
                                                          ^
/home/pierre/wrk/browsercore/src/dom/text.zig:29:41: 0xc6c4fd in get_wholeText (browsercore-wpt)
        return try parser.textWholdeText(self);
                                        ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/generate.zig:618:13: 0xb92fde in getter (browsercore-wpt)
            const res = @call(.auto, getter_func, args);
            ^
../../../../v8/src/api/api-arguments-inl.h:201:3: 0x18d8bb0 in BasicCallNamedGetterCallback (../../../../v8/src/objects/objects.cc)
../../../../v8/src/api/api-arguments-inl.h:315:10: 0x18b0777 in CallAccessorGetter (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:1442:34: 0x18aed19 in GetPropertyWithAccessor (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:1185:16: 0x18ad843 in GetProperty (../../../../v8/src/objects/objects.cc)
../../../../v8/src/ic/ic.cc:507:5: 0x291f528 in Load (../../../../v8/src/ic/ic.cc)
../../../../v8/src/ic/ic.cc:2675:3: 0x292e8ce in __RT_impl_Runtime_LoadNoFeedbackIC_Miss (../../../../v8/src/ic/ic.cc)
../../../../v8/src/ic/ic.cc:2660:1: 0x292e3f7 in Runtime_LoadNoFeedbackIC_Miss (../../../../v8/src/ic/ic.cc)
???:?:?: 0x11e85fe in ??? (???)
Unwind error at address `:0x11e85fe` (error.AddressOutOfRange), trace may be incomplete

???:?:?: 0x161da91 in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0x7fa44009e3fa in ??? (???)
???:?:?: 0x7fa4400a237c in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0xe4a69b in ??? (???)
???:?:?: 0xe4a3c6 in ??? (???)
../../../../v8/src/execution/simulator.h:155:12: 0x1c58cfa in Invoke (../../../../v8/src/execution/execution.cc)
../../../../v8/src/execution/execution.cc:538:10: 0x1c59d7c in CallScript (../../../../v8/src/execution/execution.cc)
../../../../v8/src/api/api.cc:2272:7: 0xd9771d in Run (../../../../v8/src/api/api.cc)
/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp:414:54: 0xcd3f53 in v8__Script__Run (/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp)
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/v8.zig:1679:30: 0xb0e8ab in run (browsercore-wpt)
        if (c.v8__Script__Run(self.handle, ctx.handle)) |value| {
                             ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:449:28: 0xb0f405 in exec (browsercore-wpt)
        const res = scr.run(context) catch {
                           ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:328:21: 0xacf357 in run (browsercore-wpt)
        try res.exec(alloc, script, name, self.isolate, self.context.?, try_catch);
                    ^
/home/pierre/wrk/browsercore/src/wpt/run.zig:136:16: 0xacf088 in evalJS (browsercore-wpt)
    try env.run(alloc, script, name, &res, null);
               ^
/home/pierre/wrk/browsercore/src/wpt/run.zig:99:25: 0xad3d58 in run__anon_100229 (browsercore-wpt)
        res = try evalJS(js_env, alloc, src, "");
                        ^
/home/pierre/wrk/browsercore/src/main_wpt.zig:126:28: 0xadd412 in main (browsercore-wpt)
        const res = wpt.run(&arena, apis, wpt_dir, tc, &loader) catch |err| {
                           ^
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/std/start.zig:585:37: 0xae0fa6 in main (browsercore-wpt)
            const result = root.main() catch |err| {
                                    ^

make: Nothing to be done for 'tests/wpt/dom/nodes/Node-cloneNode.html'.

Add CLI argument to specify an HTML local file path to load

For main and shell commands

lexbor basic usage from Zig

Choose an HTML parser lib

Ecrire un parser HTML est une tâche complexe :

il faut gérer les standards ( HTML5 et les versions plus anciennes)
il faut gérer les erreurs HTML courantes pour supporter un maximum de sites webs
il faut un parser rapide et léger, ce qui suppose des algorithmes assez complexes

Cette tâche a été réalisée de nombreuses fois, par conséquent l'idée n'est pas de ré-implémenter un parser from scratch (au moins dans un premier temps), mais de ré-utiliser une lib existante.

Les critères de choix pour cette lib :

Standard et avec une bonne gestion des erreurs pour couvrir un maximum de sites. Idéalement la lib doit passer les tests html5lib
Bonnes performances
Facile à intégrer en Zig. Donc idéalement en Zig, ou a défaut en C. C++ et Rust sont envisageables également mais avec plus de travail pour réaliser des bindings.
Bien sûr les critères de choix standards : maintenue, populaire, ancienneté, dévelopeurs, license.

HTTP webserver

Comment properties and methods

https://dom.spec.whatwg.org/#interface-comment

EventTarget properties and methods

Depends on #36

Element properties and methods

WebIDL parser and checker

On pourrait utiliser WebIDL afin de s’assurer:

que l’ensemble des méthodes du standard DOM sont implémentées
que la signature de ces méthodes est conforme

Tous les navigateurs modernes se basent sur WebIDL et plusieurs parsers existent déjà, souvent en Python.

L’idée serait pour chaque fichier WebIDL présent de:

parser le fichier
vérifier si une implémentation zig existe et si toutes les propriétés (getter/setter) et méthodes sont présentes et ont la signature conforme au standard
si ce n’est pas le cas générer une erreur avec les signatures attendues

Ce parsing pourrait se faire comptime.

Je le vois à la fois comme un outil de conformité et comme un outil de développement des Web APIs puisque pour chaque nouvel API le dev workflow pourrait commencer par ajouter le fichier WebIDL correspondant et ainsi générer la structure de l’implémentation (toutes les fonctions et leurs signatures).

CharacterData properties and methods

HTMLElement properties and methods

https://html.spec.whatwg.org/#htmlelement

specifics like anchor...

DOM document.createElement()

Zig Segfault when building with syntax error

When applying a patch with a basic syntax error (here an extra parameter on a fonction call) after calling jsruntime eval js, the zig build crashes with a segfault on zig 0.11.0.

If the same syntax is done before jsruntime call, there is no segfault.

The patch

diff --git a/src/main_wpt.zig b/src/main_wpt.zig
index f6d5624..4703389 100644
--- a/src/main_wpt.zig
+++ b/src/main_wpt.zig
@@ -135,7 +135,7 @@ pub fn main() !void {
         };
         // no need to call res.deinit() thanks to the arena allocator.
 
-        const suite = try Suite.init(alloc, tc, res.success, res.result, res.stack);
+        const suite = try Suite.init(alloc, tc, res.success, res.result, res.stack, true);
         try results.append(suite);
 
         if (out == .json) {

The crash

$ make wpt
Building wpt...
zig build-exe browsercore-wpt Debug native: error: the following command terminated unexpectedly:
/usr/local/zig-0.11.0/zig build-exe /home/pierre/wrk/browsercore/src/main_wpt.zig /home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/v8/x86_64-linux/debug/libc_v8.a /home/pierre/wrk/browsercore/vendor/lexbor/liblexbor_static.a /home/pierre/wrk/browsercore/vendor/libiconv/lib/libiconv.a /home/pierre/wrk/browsercore/vendor/netsurf/lib/libdom.a /home/pierre/wrk/browsercore/vendor/netsurf/lib/libhubbub.a /home/pierre/wrk/browsercore/vendor/netsurf/lib/libparserutils.a /home/pierre/wrk/browsercore/vendor/netsurf/lib/libwapcaplet.a /home/pierre/wrk/browsercore/vendor/netsurf/wrapper/wrapper.c -lc++ --cache-dir /home/pierre/wrk/browsercore/zig-cache --global-cache-dir /home/pierre/.cache/zig --name browsercore-wpt --mod jsruntime_build_options::/home/pierre/wrk/browsercore/zig-cache/c/9d8fb2d4b55029931e2bde5400d09d9a/options.zig --mod jsruntime:jsruntime_build_options,tigerbeetle-io,v8:/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/api.zig --mod v8::/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/v8.zig --mod tigerbeetle-io::/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/tigerbeetle-io/io.zig --deps jsruntime -I /home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src -I /home/pierre/wrk/browsercore/vendor/lexbor-src/source -I /home/pierre/wrk/browsercore/vendor/libiconv/include -I /home/pierre/wrk/browsercore/vendor/netsurf/libdom/src -I /home/pierre/wrk/browsercore/vendor/netsurf/libhubbub/src -I /home/pierre/wrk/browsercore/vendor/netsurf/libparserutils/src -I /home/pierre/wrk/browsercore/vendor/netsurf/libwapcaplet/src -I /home/pierre/wrk/browsercore/vendor/netsurf/include -I /home/pierre/wrk/browsercore/vendor/netsurf/wrapper --listen=- 
Build Summary: 6/11 steps succeeded; 1 failed (disable with --summary none)
wpt transitive failure
└─ run browsercore-wpt transitive failure
   ├─ zig build-exe browsercore-wpt Debug native failure
   └─ install transitive failure
      └─ install browsercore-wpt transitive failure
         └─ zig build-exe browsercore-wpt Debug native (+1 more reused dependencies)
error: the following build command failed with exit code 1:
/home/pierre/wrk/browsercore/zig-cache/o/17ca63e61969c2fc9dd25230de58929f/build /usr/local/zig-0.11.0/zig /home/pierre/wrk/browsercore /home/pierre/wrk/browsercore/zig-cache /home/pierre/.cache/zig wpt -Dengine=v8 --
Build ERROR
make: *** [Makefile:52: wpt] Error 1

Text properties and methods

Chrome extension support

DOM tests utils

497a111

Merge shell in main build?

shell usage is for now a separate build and a separate CLI browsercore-shell.

We could merge it in the main build and use it as a optional argument, ie.

browsercore <arg> to launch the main server
browsercore --shell <arg> to launch the shell

It makes more sense as a CLI usage perspective (only 1 binary) and simplify the build process.

The only downside is that it adds linenoise as a build requirement, add therefore increase the binary size. But I don't think it's a big problem, as linenoise is meant to be small.
We can also add a build option --no-shell (non-default) to keep the option to build from sources linenoise dependancy and without shell support.

CDP server

Create a simple server listening on a Unix socket for CDP messages:

read incoming CDP commands
write corresponding CDP responses

The server should be able to use the jsruntime I/O loop to keep a single-threaded architecture while supporting async commands.

WebComponents support

NetSurf Experimentation

NetSurf est un navigateur indépendant (initialement développé pour RISC OS) des années 2000-2010, toujours maintenu mais sans développement actif.

Il présente plusieurs avantages:

une structure modulaire qui permet d’utiliser séparament les différents composants
en particulier une API du DOM “pure” c’est à dire sans rendering, layout, painting, ni implémentation Javascript
Une couverture des standards qui semble bonne, en tout cas côté DOM (Level Core 3) et du parser (HTML5)
Légerté et performance sont dans les objectifs du projet (à voir en pratique)
Codé en C ce qui devrait faciliter l’embedding
Un license souple pour les libs (MIT)

On utilise actuellement Lexbor qui présente en théorie certains avantages:

code plus récent, développement plus actif
focus important sur les performances
pas de dépendances

Cependant la couverture plus importante des APIs du DOM dans le projet NetSurf permettrait d’avoir plus rapidement une version beta.

L’idée est donc de tester une intégration de NetSurf à la place de Lexbor.

Add build/dev/test instructions for jsruntime and browsercore

Dynamic JS execution

Un browser classique, y compris Chrome Headless, part du principe que l’ensemble du Javascript inclus dans un site web est nécessaire à son correct rendu. Par conséquent il execute l’ensemble du Javascript inclus. Ce JS peut être coûteux, en CPU/RAM mais aussi en réseau (calls Ajax).

Dans le cas d’un browser headless sans rendering graphique on pourrait partir du principe que seul le Javascript impliqué dans les actions pilotées par CDP est essentiel.

Par exemple sur une page produit d’un site e-commerce, on pourrait vouloir scraper les notes et commentaires des utilisateurs mais pas les produits similaires (2 cas classiques utilisant du Javascript). Dans ce cas pourquoi charger le JS des produits similaires si celui-ci ne va pas être utilisé ?

Par ailleurs cette execution dynamique (ou partielle) est aussi un piège pour browser fingerprinting, voire des résultats incohérents du JS suite à cette execution partielle.

2 options à première vue:

Une heuristique (ou une IA?) basée sur chaque script CDP (et chaque page web), donc en dehors du browser lui-même
Un principe d’execution séquentielle. On execute rien dans un premier temps, si le code CDP ne trouve pas un élément on avance d’une étape en JS, et ainsi de suite. Pas sûr que cette technique soit si efficace compte tenu de l’objectif d’économie initial.

Le principe est simple mais la mise en place d’une telle fonctionnalité est très complexe, car 2 éléments de nature différentes sont concernées:

le code JS du site, décidé côté serveur (même si il s’execute côté client)
le code CDP du client

Le code JS n’a pas idée du code CDP, et le code CDP ne connait pas le détail du code JS, il s’intéresse juste au résultat (à la différence de la retro-ingénierie en HTTP de base).

Use an existing DOM API test suite to ensure correctness

There is also xml tests cases on the libdom lib, maybe we can also leverage that (libdom.it - Document Object Model Library)

I don’t know if they have some links with the web platform tests.

We can test the browser against the web plateform tests.
https://web-platform-tests.org/index.html

The tests are standardized, it consists to a load an html file and it gives result.
Example: https://wpt.live/dom/attributes-are-nodes.html

Tests are grouped by RFC: for dom: https://wpt.live/dom/

‌

We can easily compare with major browsers results: https://wpt.fyi/results/dom?label=experimental&label=master&aligned

Servo uses them to track its improvements: https://wpt.servo.org/

The last thing: it’s a good metricsto show the browser capabilities in detail to users.

constructor: global document requirement

Some types defined in the DOM specs require global document access to be properly created.

The new Document() constructor steps are to set this’s origin to the origin of current global object’s associated Document. [HTML]

https://dom.spec.whatwg.org/#dom-document-document

The new DocumentFragment() constructor steps are to set this’s node document to current global object’s associated Document.

https://dom.spec.whatwg.org/#dom-documentfragment-documentfragment

The new Text(data) constructor steps are to set this’s data to data and this’s node document to current global object’s associated Document.

https://dom.spec.whatwg.org/#dom-text-text

The new Comment(data) constructor steps are to set this’s data to data and this’s node document to current global object’s associated Document.

https://dom.spec.whatwg.org/#dom-comment-comment

netsurf: rename stringFromData and stringToData into strFromdata and strToData

          I think we could try to shorter those functions as they are used a lot, like `strFromData` and `strToData`. It's not a lot but still 3 chars less and it's as clear as the full version I think.

Let's do that in a separate PR.

Originally posted by @francisbouvier in #100 (comment)

Add option to filter test

When working on a branch it's convenient to run test only for this case at debug stage, before running all tests.

Relates to lightpanda-io/zig-js-runtime#147

node.ReplaceChild with null should throw a TypeError

According to the test https://wpt.live/dom/nodes/Node-replaceChild.html
passing a null to replaceChild must throw a TypeError.

test(function() {
  var a = document.createElement("div");
  assert_throws_js(TypeError, function() {
    a.replaceChild(null, null);
  });

  var b = document.createElement("div");
  assert_throws_js(TypeError, function() {
    a.replaceChild(b, null);
  });
  assert_throws_js(TypeError, function() {
    a.replaceChild(null, b);
  });
}, "Passing null to replaceChild should throw a TypeError.")

Currently, browsercore crashes.

#
# Fatal error in ../../../../v8/src/api/api-inl.h, line 130
# Debug check failed: that == nullptr || v8::internal::Object( *reinterpret_cast<const v8::internal::Address*>(that)) .IsJSReceiver().
#
#
#
#FailureMessage Object: 0x7ffefb567840
==== C stack trace ===============================

    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbf3e73]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbf200d]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xc8b172]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xc8ac05]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xcc8c56]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbef04b]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbecaa0]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xb5ab29]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xb5aa38]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbd9313]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xbc2285]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xb91ba6]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0xb3cf33]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0x262f03a]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0x262d9f0]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0x262c151]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0x262bc17]
    /home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test() [0x10ff4bf]
run test: error: while executing test 'test_0', the following command terminated with signal 5 (expected exited with code 0):
/home/pierre/prs/brwsrcr/browsercore/zig-cache/o/1790b113cc59701313ba7b8f552a0e82/test --listen=-

use only one command to build everything from sources zig build
cross-compilation

The goal of this task is:

be able to parse and validate a css query
match the query against the dom node tree to find corresponding node

This work might be implementde in pure zig.
Possible inspiration:

https://github.com/andybalholm/cascadia

See slack discussion: https://lightpanda.slack.com/archives/C05TRU6RBM1/p1702481851141279

dom: characterDATA: get_data crash

$ make wpt --  tests/wpt/dom/nodes/Node-nodeValue.html
Building wpt...
thread 62649 panic: attempt to use null value
/home/pierre/wrk/browsercore/src/netsurf.zig:746:70: 0xc308c1 in characterDataData (browsercore-wpt)
    const err = characterDataVtable(cdata).dom_characterdata_get_data.?(cdata, &s);
                                                                     ^
/home/pierre/wrk/browsercore/src/dom/character_data.zig:58:44: 0xc30a7d in get_data (browsercore-wpt)
        return try parser.characterDataData(self);
                                           ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/generate.zig:618:13: 0xb8a21e in getter (browsercore-wpt)
            const res = @call(.auto, getter_func, args);
            ^
../../../../v8/src/api/api-arguments-inl.h:201:3: 0x18d8bb0 in BasicCallNamedGetterCallback (../../../../v8/src/objects/objects.cc)
../../../../v8/src/api/api-arguments-inl.h:315:10: 0x18b0777 in CallAccessorGetter (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:1442:34: 0x18aed19 in GetPropertyWithAccessor (../../../../v8/src/objects/objects.cc)
../../../../v8/src/objects/objects.cc:1185:16: 0x18ad843 in GetProperty (../../../../v8/src/objects/objects.cc)
../../../../v8/src/ic/ic.cc:507:5: 0x291f528 in Load (../../../../v8/src/ic/ic.cc)
../../../../v8/src/ic/ic.cc:2675:3: 0x292e8ce in __RT_impl_Runtime_LoadNoFeedbackIC_Miss (../../../../v8/src/ic/ic.cc)
../../../../v8/src/ic/ic.cc:2660:1: 0x292e3f7 in Runtime_LoadNoFeedbackIC_Miss (../../../../v8/src/ic/ic.cc)
???:?:?: 0x11e85fe in ??? (???)
Unwind error at address `:0x11e85fe` (error.AddressOutOfRange), trace may be incomplete

???:?:?: 0x161da91 in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0xe51e8d in ??? (???)
???:?:?: 0xe4a69b in ??? (???)
???:?:?: 0xe4a3c6 in ??? (???)
../../../../v8/src/execution/simulator.h:155:12: 0x1c58cfa in Invoke (../../../../v8/src/execution/execution.cc)
../../../../v8/src/execution/execution.cc:538:10: 0x1c59d7c in CallScript (../../../../v8/src/execution/execution.cc)
../../../../v8/src/api/api.cc:2272:7: 0xd9771d in Run (../../../../v8/src/api/api.cc)
/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp:414:54: 0xcd3f53 in v8__Script__Run (/home/pierre/prs/brwsrcr/jsruntime-lib/vendor/zig-v8/src/binding.cpp)
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/vendor/zig-v8/src/v8.zig:1679:30: 0xb0e8ab in run (browsercore-wpt)
        if (c.v8__Script__Run(self.handle, ctx.handle)) |value| {
                             ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:449:28: 0xb0f405 in exec (browsercore-wpt)
        const res = scr.run(context) catch {
                           ^
/home/pierre/wrk/browsercore/vendor/jsruntime-lib/src/engines/v8/v8.zig:328:21: 0xacf357 in run (browsercore-wpt)
        try res.exec(alloc, script, name, self.isolate, self.context.?, try_catch);
                    ^
/home/pierre/wrk/browsercore/src/wpt/run.zig:136:16: 0xacf088 in evalJS (browsercore-wpt)
    try env.run(alloc, script, name, &res, null);
               ^
/home/pierre/wrk/browsercore/src/wpt/run.zig:99:25: 0xad3d58 in run__anon_100229 (browsercore-wpt)
        res = try evalJS(js_env, alloc, src, "");
                        ^
/home/pierre/wrk/browsercore/src/main_wpt.zig:126:28: 0xadd412 in main (browsercore-wpt)
        const res = wpt.run(&arena, apis, wpt_dir, tc, &loader) catch |err| {
                           ^
/usr/local/zig-0.12.0-dev.1773+8a8fd47d2/lib/std/start.zig:585:37: 0xae0fa6 in main (browsercore-wpt)
            const result = root.main() catch |err| {
                                    ^

make: Nothing to be done for 'tests/wpt/dom/nodes/Node-nodeValue.html'.