pyodide/pyodide

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.

What is Pyodide?

Pyodide is a port of CPython to WebAssembly/Emscripten.

Pyodide makes it possible to install and run Python packages in the browser with micropip. Any pure Python package with a wheel available on PyPi is supported. Many packages with C extensions have also been ported for use with Pyodide. These include many general-purpose packages such as regex, PyYAML, lxml and scientific Python packages including NumPy, pandas, SciPy, Matplotlib, and scikit-learn.

Pyodide comes with a robust Javascript ⟺ Python foreign function interface so that you can freely mix these two languages in your code with minimal friction. This includes full support for error handling, async/await, and much more.

When used inside a browser, Python has full access to the Web APIs.

Try Pyodide (no installation needed)

Try Pyodide in a REPL directly in your browser. For further information, see the documentation.

Getting Started

Pyodide offers three different ways to get started depending on your needs and technical resources. These include:

• Use a hosted distribution of Pyodide: see the Getting Started documentation.
• Download a version of Pyodide from the releases page and serve it with a web server.
• Build Pyodide from source
  • Build natively with make: primarily for Linux users who want to experiment or contribute back to the project.
  • Use a Docker image: recommended for Windows and macOS users and for Linux users who prefer a Debian-based Docker image with the dependencies already installed.

History

Pyodide was created in 2018 by Michael Droettboom at Mozilla as part of the Iodide project. Iodide is an experimental web-based notebook environment for literate scientific computing and communication.

Iodide is no longer maintained. If you want to use Pyodide in an interactive client-side notebook, see Pyodide notebook environments.

Contributing

Please view the contributing guide for tips on filing issues, making changes, and submitting pull requests. Pyodide is an independent and community-driven open-source project. The decision-making process is outlined in the Project governance.

Communication

• Blog: blog.pyodide.org
• Mailing list: mail.python.org/mailman3/lists/pyodide.python.org/
• Gitter: gitter.im/pyodide/community
• Twitter: twitter.com/pyodide
• Stack Overflow: stackoverflow.com/questions/tagged/pyodide

License

Pyodide uses the Mozilla Public License Version 2.0.

pyodide's Issues

Raise exceptions from Python

Right now, exceptions are returned as strings. But they should actually be "thrown" on the Javascript side.

Fix and/or document issues around finalization of objects

From the embind documentation:

Embind has support for binding most C++ constructs, including those introduced in C++11 and C++14. Its only significant limitation is that it does not currently support raw pointers with complicated lifetime semantics.

This means that Python objects will "leak" under the current framework. Should investigate what, if anything, can be done there.

importation order matters for js libs / python

I wanted to attempt to run d3 in python, just to see if it was as easy as I was expecting. If I import d3 after I import python, I get the following error:

If I import d3 first (so it's in the global namespace) then python, the error does not occur.

Not sure if this is something to be filed in this repo or the other, but figured I'd put it here, since I believe pyodide grabs onto the js namespace. I'm also not sure if it is technically a bug - just wanted to put down some documentation that this is a thing.

Turn on link-time optimizations

This is a reminder to turn on link-time optimizations when emscripten-core/emscripten#6444 is merged.

Make autocompletion work

This will require making autocompletion pluggable in Iodide itself.

matplotlib widgets are ugly

Someone with some good CSS-fu could make it look much nicer.

Optimize Numpy makefile

For some as yet unknown reason, the Numpy makefile reinstalls the x86 version of Numpy 4 times before doing anything. Probably the timestamp on something unexpected is getting bumped... But it would be nice to fix, as it's kind of annoying.

META: emscripten issues

This is a meta issue to track issues filed with emscripten where I'm getting stuck.

• emscripten-core/emscripten#6308
• emscripten-core/emscripten#6309
• emscripten-core/emscripten#6310
• emscripten-core/emscripten#6355
• emscripten-core/emscripten#6362
• emscripten-core/emscripten#6444

Punt to Javascript for text encoding/decoding

The tables for some of the larger CJK encodings are quite large in the Python implementation. There should be no need for this, since the browser / OS already has them.

We should punt out to Javascript to handle encoding/decoding for codecs that we don't ship with Pyodide -- requiring some mapping between the Python names for the codecs and the web ones.

There is a polyfill here to handle some of the cross-browser differences.

I'd consider this very low priority. Python code should generally stick to Unicode and let the input/output layer handle encoding/decoding anyway.

Ship a font for xkcd plots

matplotlib doesn't include one by default, but expects the user to install one.

If one exists with compatible licensing, we should add it to the build.

Use WASM rather than asm.js

Hopefully this is just some compiler flags changing.

Deploy from Circle-CI

Now that CI is finally working again, it should automatically deploy to pyodide-demo, as well as deploy tagged releases somewhere useful.

Standard out

Right now, "print" doesn't work in Python. This should go to console.log and/or Iodide output.

EDIT: print is currently going to the console -- it should go to the output cell.

Move wasm_backend to matplotlib package

It's only used with matplotlib -- no need for it to be in the site-packages root.

Use Python 3.6

Currently, we use 3.5 because that was the latest known-to-work. But we should upgrade to latest stable.

Minify Javascript

pyodide.js, while very small, should probably be minimized when copied to the build directory.

Tensorflow wasm module with python bindings

Make work on Chrome

Use something other than `embind.h` for C++/Javascript interop

embind isn't currently maintained (see emscripten-core/emscripten#6362 (comment)), and likely is not the most efficient use of resources.

It would be great to make a lower-level pure C way to interface with Javascript objects and use that instead (largely based on embind's existing wire protocol).

"import numpy" runs out of memory on Chrome

"Error: Traceback (most recent call last): File "<string>", line 1, in <module> File "/lib/python3.6/site-packages/lazy_import.py", line 124, in __getattribute__ _load_module(self) File "/lib/python3.6/site-packages/lazy_import.py", line 493, in _load_module msg.format(**modclass._lazy_import_error_strings)) from None ImportError: sitecustomize attempted to use a functionality that requires module numpy, but it couldn't be loaded. Please install numpy and retry. at emval_allocator_1 (eval at craftEmvalAllocator (http://127.0.0.1:8000/pyodide.asm.js:1:2949151), <anonymous>:7:11) at __emval_new (http://127.0.0.1:8000/pyodide.asm.js:1:2949581) at RG (http://127.0.0.1:8000/pyodide.asm.js:19:22381) at VG (http://127.0.0.1:8000/pyodide.asm.js:19:27159) at Array.asm.__Z9runPythonNSt3__212basic_stringIwNS_11char_traitsIwEENS_9allocatorIwEEEE (http://127.0.0.1:8000/pyodide.asm.js:36:618068) at ftCall_vii (http://127.0.0.1:8000/pyodide.asm.js:1:3993583) at odc (http://127.0.0.1:8000/pyodide.asm.js:10:141771) at $F (http://127.0.0.1:8000/pyodide.asm.js:19:1710) at asm.__ZN10emscripten8internal7InvokerINS_3valEJNSt3__212basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEEEE6invokeEPFS2_S9_EPNS0_11BindingTypeIS9_EUt_E (http://127.0.0.1:8000/pyodide.asm.js:36:705906) at Object.runPython (eval at new_ (http://127.0.0.1:8000/pyodide.asm.js:1:2931430), <anonymous>:8:10)"

Parallel builds

Pyodide doesn't currently build correctly with -j. It would be nice to have that working.

IPython magic syntax

I'm not 100% sold we want to make this work, but we should at least have a placeholder for discussion.

Many of the standard magics don't work in this context, but I think the real point is whether supporting the ones that do (timeit, for example) should be done through magics or some other more standard Python mechanism.

Handle passing more than 5 arguments when calling JS from Python

Right now the code just hardcodes 1, 2, 3, 4, and 5 arguments, but some modern C++ varargs feature should hopefully remove that necessity. (see jsproxy.cpp)

numpy.fft doesn't compile when WASM=1

All other modules are working except this one. Since it's kind of a corner use case for the kinds of data science I anticipate, I'm punting on this for now. But it would be nice to resolve eventually.

Python objects "leak" when passed to Javascript

Since there is no concept of a finalizer in Javascript, there is no way to decrement the Python object's reference count when it goes out of Javascript scope or is collected by the gc.

One solution is to add such a feature to Javascript (lol).

Another might be to pass weak references to Javascript and just let Python deal with lifetimes alone, and accessing a dead object from Javascript would raise an exception.

Keyboard events don't work in matplotlib

The canvas for the plot doesn't ever seem to get keyboard focus. I think this is due to how React handles keyboard events. Need to find a solution.

Improve parsing of result line

The parsing of the input Python to find the last line which will be evaluated (rather than executed) to provide the result is probably a little brittle in certain corner cases. We should look at what IPython does here and copy that.

TypeError: window[languageModule][evaluator] is not a function

I tried https://iodide-project.github.io/pyodide-demo/python.html and get this:

click to scroll this output

TypeError: window[languageModule][evaluator] is not a function
evaluateCodeCell/<@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:119823:16
createThunkMiddleware/</</<@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:75401:18
dispatch@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:56457:18
evaluateCell/<@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:119983:7
createThunkMiddleware/</</<@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:75401:18
dispatcher[action]@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:119175:41
callback@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:119197:5
boundFunc@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:37666:9
ReactErrorUtils.invokeGuardedCallback@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:37672:7
executeDispatch@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:37456:5
executeDispatchesInOrder@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:37479:5
executeDispatchesAndRelease@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:15214:5
executeDispatchesAndReleaseTopLevel@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:15225:10
forEachAccumulated@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:45314:5
processEventQueue@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:15425:7
runEventQueueInBatch@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:60933:3
handleTopLevel@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:60943:5
handleTopLevelImpl@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:64187:5
perform@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:16592:13
batchedUpdates@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:64106:14
batchedUpdates@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:12320:10
dispatchEvent@https://iodide-project.github.io/pyodide-demo/iodide.dev.js:64262:7

Download

Is there a prebuilt version I can download?

Start a "standard library"

It would be useful to have a standard library for pyodide that gives Pythonic access to browser services.

Most important to start would be I/O, and bonus points if it can mimic an existing interface, like urllib.

Use FORCE_ALIGNED_MEMORY

Python and Numpy are both reportedly alignment clean, or at least were at one time when they worked on Sparc. Setting this option should give us a speedup.

https://github.com/kripken/emscripten/blob/master/src/settings.js#L103

Run in node.js

I don't really understand the implications of this or whether it's possible, but it might be handy for testing.

Hardcode font cache for matplotlib

Since there's no "persistent filesystem", the font cache is regenerated on every import of matplotlib. Since we only allow access to the builtin matplotlib fonts anyway, the cache should never change, and we should be able to just ship a hardcoded copy, speeding up import times.

Make package fetching happen automatically

Right now, to load a package (a Python library that isn't in the core distribution), you have to add a Javascript cell with:

pyodide.loadPackage('numpy')

It would be great if all it took was (from Python):

import numpy

This is tricky, because the fetching must happen asyncronously, so you have to have a way to hop back to a currently running code cell after fetching the content. I think this will require some shenanigans with the embedded Python interpreter, which doesn't seem to have a sort of "pause and resume" API. But maybe I just haven't found it.

should data club demo run-all?

i know it was not really designed explicitly to work with run all (it's really meant as a cell-by-cell demo), but is the data club demo here:
https://iodide.io/pyodide-demo/data-club-demo-2018-05-29.html
an appropriate test case for async stuff?

numpy fails to import

I can't import numpy, has anyone ever had this problem? am i doing anything wrong?
Browser: Chrome Version 67.0.3396.87 (Official Build) (64-bit)
My code

languagePluginLoader.then(() => {
    // pyodide is now ready to use...
   pyodide.loadPackage('numpy').then(() => {
        pyodide.runPython('import numpy as np');
       
  });
});

the error

Error: Traceback (most recent call last):
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.


    at _hiwire_throw_error (https://iodide.io/pyodide-demo/pyodide.asm.js:4:34764)
    at wasm-function[330]:1321
    at wasm-function[333]:621
    at Object.Module.__runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:1853747)
    at Object.Module.runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:37861)
    at t (https://iodide.io/dist/iodide.pyodide-20180605.js:12:191465)
    at https://iodide.io/dist/iodide.pyodide-20180605.js:110:12423
    at dispatch (https://iodide.io/dist/iodide.pyodide-20180605.js:46:27803)
    at code.cellType.p.then (https://iodide.io/dist/iodide.pyodide-20180605.js:12:191240)

Organize code into separate files

It's all one big .cpp file that should be organized by type.

Scientific stack

In order of priority:

• Numpy
• Pandas
• Scipy (possibly just parts)
• matplotlib

Modularization

This is a big one.
It would be nice to not have to build all libraries into one monolithic .data file, but to have individual libraries download on demand. This might take the form of an "overlay filesystem" (a general emscripten solution), or a custom import hook (a Python-specific solution).
Also, modularizing the build system to make it easier to add new libraries would be nice. Bonus points to build such a tool on top of Anaconda.

Wrap Python objects for Javascript usage

It should be possible to write an embind wrapper so Javascript can do things with Python objects, such as call methods, get items etc.

Attach traceback and other metadata to Python exception when returned to Javascript

Packaging

Adding a new package to pyodide involves editing many disjointed files.

Instead, each package should be defined as a set of independent "recipes" that the build system could pick up and build everything needed, including the package dependency graph.

conda comes pretty close to what is needed here, and could be adapted to support emscripten. However, it could be overkill. Research should be done.

Provide conveniences around converting Numpy arrays to/from Javascript typed arrays

Currently, you can pass a typed array to Python, getting a buffer which you can then use with np.frombuffer to get a Numpy array (and make sure you match the datatype on both sides). When passing a Numpy array to Javascript, you get regular (non-typed) Javascript arrays, which is a less-than-optimal use of memory. This requires getting the datatype correct and knowing how to do this.

It would be nice to have a more convenient way to do this. It can't be built-in, since the user may not be using Numpy, but maybe if there's an explicit way to turn this on, the conversion code could then implicitly do this.

Compared to pypy.js

Would anyone happen to know about pypy.js and be able to compare pyodide and pypy.js? Which is a more complete, functioning implementation of Python?

At least Pyodide is alive, unlike pypy.js, but I want to know as of now which is more complete. And, if pypy.js is more complete, how quick is Pyodide coming up to speed, and, unassumingly, surpassing pypy.js?

Fix NaN compiler warnings

emcc: warning: cannot represent a NaN literal '0x3fd5f70' with custom bit pattern in NaN-canonicalizing JS engines (e.g. Firefox and Safari) without erasing bits!
  in   %. = select i1 %19, double 0xFFF8000000000000, double 0x7FF8000000000000 in _PyFloat_Unpack2() 
warning: unexpected return type i32 in call to '_PyTraceMalloc_Fini', should be void
emcc: warning: cannot represent a NaN literal '0x3fd5f70' with custom bit pattern in NaN-canonicalizing JS engines (e.g. Firefox and Safari) without erasing bits!
  in   %. = select i1 %60, double 0xFFF8000000000000, double 0x7FF8000000000000 in _Py_parse_inf_or_nan() 

Run Python unit tests

We should have a mode to include and run all of the Python unit tests to see where we stand on things working.

matplotlib figure doesn't update when elements are changed

It's somehow not triggering the call to draw_idle like it should...

Separate building wasm from filesystem

Right now we do it together, but it would speed development to separate it out.

numpy not working in iframe context

@mdboom -- this is probably a no-op for now; i will supply additional details when the iframe branch is a bit further along

but for reference (and also for @hamilton) in the iframe-2 branch, when i load
http://localhost:8888/dev/iodide.editor.dev.html?url=https://iodide.io/pyodide-demo/data-club-demo-2018-05-29.html
and execute cell-by-cell, all the base python demo stuff works as expected, and when i run pyodide.loadPackage('numpy') i ge the expected "loaded numpy" message; but when i try to run the numpy sine wave cell, i get:

Error: Traceback (most recent call last):
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.


    at _hiwire_throw_error (https://iodide.io/pyodide-demo/pyodide.asm.js:4:34764)
    at wasm-function[330]:1321
    at wasm-function[333]:621
    at Object.Module.__runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:1853747)
    at Object.Module.runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:37861)
    at http://localhost:8888/dev/iodide.eval-frame.dev.js:13574:49
    at http://localhost:8888/dev/iodide.eval-frame.dev.js:79279:18
    at dispatch (http://localhost:8888/dev/iodide.eval-frame.dev.js:52298:18)
    at evaluationQueue.then (http://localhost:8888/dev/iodide.eval-frame.dev.js:13732:52)

no need to worry about this just yet, when i have a sharable link to that branch i'll update this thread (of course, you're always welcome to check out iframe-2 and poke around if you're curious, but it's still in rough shape for now)

More refined object sharing

I'm starting to feel like the shared namespace between Python and JS creates more problems than it solves, and a more explicit way of managing this would be better.

I would propose something like (in Python):

from js import window

and in Javascript:

var sys = pyimport('sys')

The idea is that each side would have their own namespaces, which is persisted between cells (as notebook users are used to), but to get objects between the two worlds, they would have to use an import statement/function of some kind.