jupyterlite/xeus-lua-kernel

A Lua kernel for JupyterLite, powered by Xeus

jupyterlite-xeus-lua-kernel

A Lua kernel for JupyterLite running in the browser.

Requirements

• JupyterLite >= 0.1.0b16

Install

To install the extension, execute:

pip install jupyterlite_xeus_lua

Then build your JupyterLite site:

jupyter lite build

Uninstall

To remove the extension, execute:

pip uninstall jupyterlite_xeus_lua

Contributing

Development install

Note: You will need NodeJS to build the extension package.

The jlpm command is JupyterLab's pinned version of yarn that is installed with JupyterLab. You may use yarn or npm in lieu of jlpm below.

# Clone the repo to your local environment
# Change directory to the jupyterlite_xeus_lua directory
# Install package in development mode
python -m pip install -e .

# Link your development version of the extension with JupyterLab
jupyter labextension develop . --overwrite

# Rebuild extension Typescript source after making changes
jlpm run build

You can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.

# Watch the source directory in one terminal, automatically rebuilding when needed
jlpm run watch
# Run JupyterLab in another terminal
jupyter lab

With the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).

Development uninstall

pip uninstall jupyterlite_xeus_lua

In development mode, you will also need to remove the symlink created by jupyter labextension develop command. To find its location, you can run jupyter labextension list to figure out where the labextensions folder is located. Then you can remove the symlink named jupyterlite-xeus-lua within that folder.

Packaging the extension

See RELEASE

xeus-lua-kernel's Issues

Bundle the wasm file

now that we move the xeus lua related files to the asset dir via https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/3a22e56672ef5aca0655fc64aaa39fac74e5b6a2/webpack.config.js#L4

we are facing the problem that xeus_lua.js it not found

ModuleNotFoundError: Module not found: Error: Can't resolve './xeus_lua' in '/home/derthorsten/src/jupyterlite_xeus_lua/lib'
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/Compilation.js:1930:28
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:793:13
    at eval (eval at create (/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:275:22
    at eval (eval at create (/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:431:22
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:124:11
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:665:25
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:850:8
    at /home/derthorsten/src/jupyterlite_xeus_lua/node_modules/webpack/lib/NormalModuleFactory.js:970:5
resolve './xeus_lua' in '/home/derthorsten/src/jupyterlite_xeus_lua/lib'
...
...
...

@jtpio you mentioned:

Locally excluding the emscripten generated files seems to help build the extension.

However the files are not properly emitted to labextension/static yet.

@jtpio any hint on how to continue on that?

Do we need to change the import statement?
https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/3a22e56672ef5aca0655fc64aaa39fac74e5b6a2/src/worker.ts#L3

this is all a bit beyond my typescript / webpack competence ...

Reducing the size of the wasm file

The current wasm file is about 40MB which is quite huge.

Wondering whether there would be a way to reduce the size. Maybe that would involve passing a couple of flags to the emscripten build command.

SharedArrayBuffer

@jtpio here a few infos on sharedarraybuffers.

https://developers.google.com/search/blog/2021/03/sharedarraybuffer-notes
https://web.dev/cross-origin-isolation-guide/

Not sure what the cross-origin isolation thingy actually means and if we could use it to keep the SharedArrayBuffer

how to setup kernel logo properly

@jtpio any hint on how to add the lua logo for the kernel properly

minify generated js files

this is a hint on how to minimize the emscripten generated js files
emscripten-core/emscripten#8773

`build:labextension:dev` stage fails

@jtpio
here is my attempt to compile xeus-lua as seperate extension.

First I had to add a https://github.com/DerThorsten/jupyterlite_xeus_lua/blob/main/src/xeus_lua.d.ts which I had not in the latest version of xeus-lua directly integrated in jupyterlite (at some point I also had such a file in the direct integrated xeus lua)

but jlpm run build fails horrible at the build:labextension:dev stage
( jlpm run build:lib without any error)

Any hint what could be the problem?

 tsc
$ jupyter labextension build --development True .
Building extension in .

Compilation starting…


<--- Last few GCs --->

[48218:0x55a0f730ff30]   364047 ms: Scavenge 8000.4 (8222.0) -> 7996.8 (8223.0) MB, 12.5 / 0.0 ms  (average mu = 0.169, current mu = 0.038) task 
[48218:0x55a0f730ff30]   364133 ms: Scavenge 8002.9 (8223.0) -> 7996.0 (8231.2) MB, 16.2 / 0.0 ms  (average mu = 0.169, current mu = 0.038) allocation failure 
[48218:0x55a0f730ff30]   366750 ms: Mark-sweep 8011.5 (8231.2) -> 7988.5 (8231.7) MB, 2509.7 / 0.1 ms  (average mu = 0.198, current mu = 0.226) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7f3b40cc75c9 node::Abort() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 2: 0x7f3b40ba94a3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > node::SPrintFImpl<int&, char const*>(char const*, int&, char const*&&) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 3: 0x7f3b4107a832 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 4: 0x7f3b4107ab0b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 5: 0x7f3b412411d6  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 6: 0x7f3b412412c1  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 7: 0x7f3b41255c24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 8: 0x7f3b41256524 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
 9: 0x7f3b412bc5cd v8::internal::ScavengeJob::Task::RunInternal() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
10: 0x7f3b4115abdc  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
11: 0x7f3b40d44794 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
12: 0x7f3b40d47f73 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
13: 0x7f3b40230901  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
14: 0x7f3b402427fc  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
15: 0x7f3b4023102c uv_run [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/./libuv.so.1]
16: 0x7f3b40bfafd7 node::SpinEventLoop(node::Environment*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
17: 0x7f3b40d148d5 node::NodeMainInstance::Run(int*, node::Environment*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
18: 0x7f3b40d14d88 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
19: 0x7f3b40c8a9fc node::Start(int, char**) [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/../lib/libnode.so.93]
20: 0x55a0f648f0fb main [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/node]
21: 0x7f3b40293565 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
22: 0x55a0f648f146  [/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/bin/node]
An error occurred.
subprocess.CalledProcessError: Command '['node', '/home/derthorsten/src/jupyterlite_xeus_lua/node_modules/@jupyterlab/builder/lib/build-labextension.js', '--core-path', '/home/derthorsten/miniconda3/envs/jupyterlite-xeus-lua/lib/python3.9/site-packages/jupyterlab/staging', '/home/derthorsten/src/jupyterlite_xeus_lua', '--development']' died with <Signals.SIGABRT: 6>.
See the log file for details:  /tmp/jupyterlab-debug-gu5vc6u_.log

adding example notebook to build

It would be nice to add https://github.com/jupyter-xeus/xeus-lua/blob/main/notebooks/widgets.ipynb and https://github.com/jupyter-xeus/xeus-lua/blob/main/notebooks/game_of_life.ipynb to the build st we can host a demo which contains these notebooks.
I am a bit overwhelmed with https://jupyterlite.readthedocs.io/en/latest/configuring.html#adding-content

Next steps to package and distribute the kernel

Thinking about the next steps once we get the first version working as a federated extension:

1. we take the xeus lua specific parts out
2. we create a https://github.com/jupyterlite/xeus-kernel repo that other kernels like xeus-lua can depend on. This repo would contain most of the logic
3. jupyterlite-xeus-lua would live in its own repo, for example https://github.com/jupyterlite/xeus-lua-kernel, and would depend on the @jupyterlite/xeus-kernel packages. It would also have the tools (Dockerfile) to create the wasm file, and would specify it as a parameter
4. streamline (or at least document) the process to place the files generated by emscripten so they are picked up by the extension on build
5. Clean up @ts-ignore in the code

Improve string substitution in the worker file

Currently there is a build step in package.json involving a call to sed to replace a substring:

xeus-lua-kernel/package.json

Line 45 in e3a1937

"sed-worker": "shx sed -i \"s/XEUS_KERNEL_FILE/'.\\/xlua_wasm.js'/\" src/worker.ts && shx sed -i \"s/LANGUAGE_DATA_FILE/''/\" src/worker.ts ",

We should look into making this step more robust.