pyodide/pyodide-webpack-plugin

A Webpack plugin for integrating pyodide into your project.

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.

What is Pyodide?

Pyodide is a port of CPython to WebAssembly/Emscripten.

Pyodide makes it possible to install and run Python packages in the browser with micropip. Any pure Python package with a wheel available on PyPi is supported. Many packages with C extensions have also been ported for use with Pyodide. These include many general-purpose packages such as regex, PyYAML, lxml and scientific Python packages including NumPy, pandas, SciPy, Matplotlib, and scikit-learn.

Pyodide comes with a robust Javascript ⟺ Python foreign function interface so that you can freely mix these two languages in your code with minimal friction. This includes full support for error handling, async/await, and much more.

When used inside a browser, Python has full access to the Web APIs.

Try Pyodide (no installation needed)

Try Pyodide in a REPL directly in your browser. For further information, see the documentation.

Getting Started

Pyodide offers three different ways to get started depending on your needs and technical resources. These include:

• Use a hosted distribution of Pyodide: see the Getting Started documentation.
• Download a version of Pyodide from the releases page and serve it with a web server.
• Build Pyodide from source
  • Build natively with make: primarily for Linux users who want to experiment or contribute back to the project.
  • Use a Docker image: recommended for Windows and macOS users and for Linux users who prefer a Debian-based Docker image with the dependencies already installed.

History

Pyodide was created in 2018 by Michael Droettboom at Mozilla as part of the Iodide project. Iodide is an experimental web-based notebook environment for literate scientific computing and communication.

Iodide is no longer maintained. If you want to use Pyodide in an interactive client-side notebook, see Pyodide notebook environments.

Contributing

Please view the contributing guide for tips on filing issues, making changes, and submitting pull requests. Pyodide is an independent and community-driven open-source project. The decision-making process is outlined in the Project governance.

Communication

• Blog: blog.pyodide.org
• Mailing list: mail.python.org/mailman3/lists/pyodide.python.org/
• Twitter: twitter.com/pyodide
• Stack Overflow: stackoverflow.com/questions/tagged/pyodide
• Discord: Pyodide Discord

License

Pyodide uses the Mozilla Public License Version 2.0.

pyodide-webpack-plugin's Issues

Attempted import error: 'version' is not exported from 'pyodide' (imported as 'pyodideVersion').

This might be an upstream issue, but since pyodide-webpack-plugin v1.3, we are getting

Attempted import error: 'version' is not exported from 'pyodide' (imported as 'pyodideVersion').

on this line of code

import { loadPyodide, version as pyodideVersion } from 'pyodide';

I think this has to do with the fact that pyodide.js has

try{Object.assign(exports,loadPyodide)}catch(_){}

while pyodide.mjs has

export{De as loadPyodide,k as version};

I'm guessing that pyodide-webpack-plugin must have changed which of those two files it being used?

(pyodide v0.24.1)

Supporting arbitrary indexURL

So currently there is autoCdnIndexUrl option but it could also be useful to be able to pass an arbitrary indexURL, as we want to encourage people to be able to do their own deployment of the subset of packages they need.

For instance, in some cases, we may want to test the dev channel https://pyodide.org/en/stable/usage/downloading-and-deploying.html#cdn

Also how does one specify the Pyodide version to use? That would be via the version of the pyodide npm package, right? Do we expect this to work with just the latest release or a range of releases (say starting from the current one)?

Asking lots of questions as a user :)

Vue3 Tutorial Project does not build with pyodide-webpack-plugin

Hi,

I tried to use pyodide-webpack-plugin in a Vue3 Project. When I run npm run serve I get an error ReferenceError: require is not defined in ES module scope, you can use import instead from node_modules/@pyodide/webpack-plugin/plugin.mjs.

Full Error Message

> [email protected] serve
> vue-cli-service serve

 ERROR  ReferenceError: require is not defined in ES module scope, you can use import instead
ReferenceError: require is not defined in ES module scope, you can use import instead
    at Object.copy-webpack-plugin (file:///home/DABIT-ANALYTICS/sebastian.mueller/workspace/tools/tests/vue-3-tutorial/node_modules/@pyodide/webpack-plugin/plugin.mjs:84:1)
    at __webpack_require__ (file:///home/DABIT-ANALYTICS/sebastian.mueller/workspace/tools/tests/vue-3-tutorial/node_modules/@pyodide/webpack-plugin/plugin.mjs:128:40)
    at file:///home/DABIT-ANALYTICS/sebastian.mueller/workspace/tools/tests/vue-3-tutorial/node_modules/@pyodide/webpack-plugin/plugin.mjs:191:77
    at file:///home/DABIT-ANALYTICS/sebastian.mueller/workspace/tools/tests/vue-3-tutorial/node_modules/@pyodide/webpack-plugin/plugin.mjs:244:3
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
    at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
    at async Service.run (/home/DABIT-ANALYTICS/sebastian.mueller/workspace/tools/tests/vue-3-tutorial/node_modules/@vue/cli-service/lib/Service.js:247:5)

Reproduce

Scaffold a vue3 Project using

npm install -g @vue/cli@next
vue create vue-3-tutorial

(default vue3 configuration)
Add a vue.confi.mjs with

import PyodidePlugin from "@pyodide/webpack-plugin";
import {defineConfig} from '@vue/cli-service';

export default defineConfig({
    configureWebpack: () => {return {plugins: [new PyodidePlugin()],}},
    transpileDependencies: true
})

and delete the vue.config.js.

Update devDependencies of the package.json to be

{
    "@pyodide/webpack-plugin": "^1.0.0-alpha.1",
    "@babel/core": "^7.12.16",
    "@babel/eslint-parser": "^7.12.16",
    "@vue/cli-plugin-babel": "~5.0.0",
    "@vue/cli-plugin-eslint": "~5.0.0",
    "@vue/cli-service": "~5.0.0",
    "eslint": "^7.32.0",
    "eslint-plugin-vue": "^8.0.3",
    "pyodide": "^0.21.3"
  }

(e.g. add @pyodide/webpack-plugin and pyodide)

Run in the project directory

rm -rf node-modules
npm install
npm run serve

`PyodidePlugin is not a constructor` error

I'm trying to set up a webpack project with the Pyodide WebPack Plugin v1.2.0 (Pyodide 0.24.1) but I'm getting this error when starting the project:

[webpack-cli] TypeError: PyodidePlugin is not a constructor
    at module.exports (/Library/WebServer/Documents/citycompass/webpack.config.js:103:6)
    at loadConfigByPath (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/webpack-cli.js:1445:37)
    at WebpackCLI.loadConfig (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/webpack-cli.js:1515:38)
    at WebpackCLI.createCompiler (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/webpack-cli.js:1781:22)
    at Command.<anonymous> (/Library/WebServer/Documents/citycompass/node_modules/@webpack-cli/serve/lib/index.js:81:30)
    at Command.parseAsync (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/node_modules/commander/lib/command.js:935:5)
    at Command.<anonymous> (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/webpack-cli.js:1356:13)
    at Command.parseAsync (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/node_modules/commander/lib/command.js:935:5)
    at WebpackCLI.run (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/webpack-cli.js:1360:9)
    at runCLI (/Library/WebServer/Documents/citycompass/node_modules/webpack-cli/lib/bootstrap.js:9:9)
error Command failed with exit code 2.

I installed the plugin with this command:

yarn add -D pyodide @pyodide/webpack-plugin

And including it in the webpack configuration in this way:

    const PyodidePlugin = require("@pyodide/webpack-plugin");
    plugins: [
            new PyodidePlugin(),

Any clue?

Support for Pyodide 0.23.1

It appears that files to copy are only specified for versions 0.21.3 and 0.22.1.

With version 0.23.1, the following error occurs:

unable to locate './node_modules/pyodide/pyodide_py.tar' glob
unable to locate './node_modules/pyodide/pyodide.asm.data' glob

Pyodide 0.24.0 rename repodata.json to pyodide-lock.json

I got the exception about the repodata.json is missing.
pyodide/pyodide#3824

Add support for Vite

I've found it very very tricky to get Pyodide working with Vite as a bundler (which I feel has become pretty popular now) and had to give up to load things from a CDN instead.

It looks like this is what I was missing (but I couldn't find mention of it anywhere except for some random Github comment — finding any example of bundling Pyodide took me hours), but it looks like it will only work for people using Webpack.

Is it possible to develop something similar for Vite? Or is there an even simpler, plugin-free solution that I'm missing?

Sorry for the trouble and thanks in advance!

NextJS Error. Module not found

Hello, I am trying to use Pyodide with my NextJS application, and using this plugin.

This is why next.config.js

const CopyPlugin = require("copy-webpack-plugin");
const { PyodidePlugin } = require("@pyodide/webpack-plugin");

const nextConfig = {
  webpack(config, { isServer }) {
    config.plugins.push(
      new CopyPlugin({
        patterns: [
          {
            from: "node_modules/@rdkit/rdkit/dist/RDKit_minimal.wasm",
            to: "static/chunks"
          }
        ]
      })
    );

    if (!isServer) {
      config.resolve.fallback = {
        fs: false
      };
    }

    return config;
  },   
  plugins: [new PyodidePlugin()],
  output: "export",
  basePath: "/sar-in-browser",
  images: {
    unoptimized: true,
  },
};

module.exports = nextConfig;

And this is how I am trying to load pyodide:

let pyodide = await loadPyodide({
    indexURL: { indexURL: `${window.location.origin}/pyodide` },
});

Any help would be lovely!

Webpack url not found

from @msebas regarding #6

After building your fix 5d3ec27 I installed it to the test project. Trying to run the dev server using npm run serve I got an unrelated error stating that Module not found: Error: Can't resolve 'url' in ..., so I installed url using npm install --save-dev url and checked if node_modules/@pyodide/webpack-plugin was not updated by npm. Now the dev server starts up.
To do something with pyodide I updated src/main.js to be

import { createApp } from 'vue'
import App from './App.vue'
import {loadPyodide} from "pyodide";

createApp(App).mount('#app')
let pyodide = null;
async function main() {
  pyodide = await loadPyodide({ indexURL: `${window.location.origin}/pyodide` });
  return pyodide.runPython(`
    import sys
    sys.version
  `)
}
main().then(r => console.log(r));

(following the example in the README.md)

The dev-Server starts now, but loading http://localhost:8080/ leads to a console error

Uncaught (in promise) Error: Cannot find module 'http://localhost:8080/pyodide/pyodide.asm.js'
    webpackEmptyAsyncContext app.js:99
    promise callback*webpackEmptyAsyncContext app.js:98
    loadScript compat.ts:158
    loadPyodide pyodide.ts:308
    main main.js:8
    <anonymous> main.js:15
    js app.js:63
    __webpack_require__ app.js:277
    __webpack_exports__ app.js:1499
    O app.js:326
    <anonymous> app.js:1500
    <anonymous> app.js:1502
app.js:99:11

which is cause by the function

function webpackEmptyAsyncContext(req) {
	// Here Promise.resolve().then() is used instead of new Promise() to prevent
	// uncaught exception popping up in devtools
	return Promise.resolve().then(function() {
		var e = new Error("Cannot find module '" + req + "'");
		e.code = 'MODULE_NOT_FOUND';
		throw e;
	});
}

that seems to be loaded by pyodide using
loadScript = async url => await __webpack_require__("./node_modules/pyodide lazy recursive")(url)
where url is the string "http://localhost:8080/pyodide/pyodide.asm.js" (strangely calling the URL itself returns said script from the pyodide main module).

Guessing that the value of url might be not valid for webpacks resolving mechanism I dropped the indexURL from main.js (loading pyodide via pyodide = await loadPyodide();). This resulted in the same error, only the value of url changed to "webpack-internal:///./node_modules/pyodide/pyodide.asm.js".

I am a little bit at a loss what makes webpack screw up here, because the first value of url serves what is required.

PyodidePlugin is not a constructor

then I got error `PyodidePlugin is not a constructor`

pyodide version: 0.23.4
node version: 16.18.0

Critical dependency: the request of a dependency is an expression

I'm getting "Critical dependency: the request of a dependency is an expression" when building my app which I assume is triggered by the dynamic imports in some of the pyodide files. The README here suggests to use string-replace-loader but I couldn't get that to work. But I found https://stackoverflow.com/a/59235546/1976323 which suggests the following to work around the issue for now.

new webpack.ContextReplacementPlugin(/pyodide/)

It would be nice, though, if a fix was just built into this plugin someshow so that we we can just use this plugin and everything works without additional hacks.

UnhandledSchemeError

I installed this plugin as described in the readme and I started seeing the following warnings and errors when running a webpack build:

WARNING in ./node_modules/node-domexception/index.js 5:31-56
Module not found: Error: Can't resolve 'worker_threads' in '/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/node-domexception'
 @ ./node_modules/fetch-blob/from.js 3:0-44 86:16-28
 @ ./node_modules/node-fetch/src/index.js 27:0-34:28 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68
 @ ./node_modules/pyodide/pyodide.mjs
 @ ./src/index.tsx 4:0-38 7:22-33

WARNING in ./node_modules/pyodide/pyodide.mjs 9:1996-2005
Critical dependency: the request of a dependency is an expression
 @ ./src/index.tsx 4:0-38 7:22-33

WARNING in ./node_modules/pyodide/pyodide.mjs 9:2140-2149
Critical dependency: the request of a dependency is an expression
 @ ./src/index.tsx 4:0-38 7:22-33

WARNING in ./node_modules/pyodide/pyodide.mjs 9:2385-2416
Critical dependency: the request of a dependency is an expression
 @ ./src/index.tsx 4:0-38 7:22-33

4 warnings have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

ERROR in node:buffer
Module build failed: UnhandledSchemeError: Reading from "node:buffer" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:838:25
    at Hook.eval [as callAsync] (eval at create (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:835:8)
    at processResource (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:825:3)
    at NormalModule.build (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:969:15)
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/Compilation.js:1377:12
    at NormalModule.needBuild (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:1257:32)
    at Compilation._buildModule (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/Compilation.js:1358:10)
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:305:10
    at Hook.eval [as callAsync] (eval at create (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at AsyncQueue._startProcessing (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:295:26)
    at AsyncQueue._ensureProcessing (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:282:12)
    at process.processImmediate (node:internal/timers:476:21)
 @ ./node_modules/node-fetch/src/index.js 13:0-35 375:20-31 396:29-43 401:5-19 402:5-19
 @ ./node_modules/pyodide/pyodide.mjs
 @ ./src/index.tsx 4:0-38 7:22-33

ERROR in node:fs
Module build failed: UnhandledSchemeError: Reading from "node:fs" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:838:25
    at Hook.eval [as callAsync] (eval at create (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:835:8)
    at processResource (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:825:3)
    at NormalModule.build (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:969:15)
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/Compilation.js:1377:12
    at NormalModule.needBuild (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/NormalModule.js:1257:32)
    at Compilation._buildModule (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/Compilation.js:1358:10)
    at /Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:305:10
    at Hook.eval [as callAsync] (eval at create (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at AsyncQueue._startProcessing (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:295:26)
    at AsyncQueue._ensureProcessing (/Users/aaronhardy/dev/demos/pyodide-demo/node_modules/webpack/lib/util/AsyncQueue.js:282:12)
    at process.processImmediate (node:internal/timers:476:21)
 @ ./node_modules/fetch-blob/from.js 1:0-68 8:17-19 14:46-54 34:46-54 88:12-28
 @ ./node_modules/node-fetch/src/index.js 27:0-34:28 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68
 @ ./node_modules/pyodide/pyodide.mjs
 @ ./src/index.tsx 4:0-38 7:22-33

Plus several more UnhandledSchemeError errors related to node:*. Have you seen this before? I'm on [email protected], @pyodide/[email protected], and [email protected].

Option to choose list of files to be copied

First of all, thanks for maintaining this project Michael, I really appreciate it.

As mentioned in pyodide/pyodide-webpack-example#7, it would be nice to test this plugin in Pyodide repository and make sure the upstream changes don't break this plugin. For that, I think we need some way to override the list of files to be copied, as upstream changes may add/remove files to be copied.

Maybe something like:

module.exports = {
  plugins: [new PyodidePlugin(
    files: ["pyodide.js", "pyodide.asm.js", "pyodide.asm.wasm"],
  )],
};