Recipes to build the packages for the emscripten-forge distribution
License: BSD 3-Clause "New" or "Revised" License
Visit emscripten-forge.org for more information and the documentation.
At the moment geopandas appears to be missing:
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
https://repo.mamba.pm/emscripten-forge/emscripte.. 14.7kB @ 30.0kB/s 0.5s
conda-forge/emscripten-32 359.0 B @ 703.0 B/s 404 failed 0.5s
https://repo.mamba.pm/emscripten-forge/noarch 441.0 B @ 807.0 B/s 0.5s
https://repo.mamba.pm/conda-forge/emscripten-32 93.0 B @ 141.0 B/s 404 failed 0.7s
https://repo.mamba.pm/conda-forge/noarch 11.0MB @ 1.4MB/s 8.0s
conda-forge/noarch 11.9MB @ 1.1MB/s 10.2s
Encountered problems while solving:
- nothing provides fiona needed by geopandas-0.10.0-pyhd8ed1ab_0
It would be great to add it to Emscripten Forge.
Pyodide has the recipe here: https://github.com/pyodide/pyodide/blob/main/packages/geopandas/meta.yaml
Looks like we might need to package a few others before such as shapely.
Some packages contain a hard-coded python version. We need to change this st. #303 will work fine.
Some of these packages are:
I have a few packages I build with emscripten (arrow, exprtk), would love to have a unified way to maintain them.
https://github.com/timkpaine/arrow-wasm-cpp
https://github.com/timkpaine/exprtk.js
I think the package corresponds to: https://github.com/conda-forge/libxml2-feedstock/blob/main/recipe/meta.yaml
On https://repo.mamba.pm/emscripten-forge, the four links at the top (channeldata.json, linux-64, emscripten-32, noarch) all lead to "not found" errors, because the URL subdirectory /emscripten-forge is lost.
when we build the python was executable, pybind11 tries to use the wasm file as a linux64 executable in the configure step which yields strange errors
The current python build generates only broken symlinks for:
In #289 we remove these broken files
Looks like I cannot link against libsolv because of some GCC stack guard stuff.
According to this: https://stackoverflow.com/a/58873044
We should add -fno-stack-protector -U_FORTIFY_SOURCE to CXXFLAGS, CFLAGS and LDFLAGS.
peewee was recently added (#322), but it seems that an error with sqlite (added by @DerThorsten last year) is preventing us from running this small test script:
from peewee import *
# Define a database object
db = SqliteDatabase('my_database.db')
# Define a model class
class Person(Model):
name = CharField()
age = IntegerField()
class Meta:
database = db
# Create the table in the database
db.connect()
db.create_tables([Person])
person1 = Person(name='Alice', age=25)
person1.save()
This returns:
DatabaseError: database disk image is malformed
---------------------------------------------------------------------------
DatabaseError Traceback (most recent call last)
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in execute_sql(self, sql, params, commit)
3235 cursor = self.cursor()
-> 3236 cursor.execute(sql, params or ())
3237 return cursor
DatabaseError: database disk image is malformed
During handling of the above exception, another exception occurred:
DatabaseError Traceback (most recent call last)
/tmp/xpython_42/2390659374.py in <cell line: 2>()
1 person1 = Person(name='Alice', age=25)
----> 2 person1.save()
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in save(self, force_insert, only)
6745 rows = self.update(**field_dict).where(self._pk_expr()).execute()
6746 elif pk_field is not None:
-> 6747 pk = self.insert(**field_dict).execute()
6748 if pk is not None and (self._meta.auto_increment or
6749 pk_value is None):
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in inner(self, database, *args, **kwargs)
1960 raise InterfaceError('Query must be bound to a database in order '
1961 'to call "%s".' % method.__name__)
-> 1962 return method(self, database, *args, **kwargs)
1963 return inner
1964
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in execute(self, database)
2031 @database_required
2032 def execute(self, database):
-> 2033 return self._execute(database)
2034
2035 def _execute(self, database):
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in _execute(self, database)
2836 self._returning = (self.table._primary_key,)
2837 try:
-> 2838 return super(Insert, self)._execute(database)
2839 except self.DefaultValuesException:
2840 pass
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in _execute(self, database)
2549 cursor = self.execute_returning(database)
2550 else:
-> 2551 cursor = database.execute(self)
2552 return self.handle_result(database, cursor)
2553
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in execute(self, query, commit, **context_options)
3242 ctx = self.get_sql_context(**context_options)
3243 sql, params = ctx.sql(query).query()
-> 3244 return self.execute_sql(sql, params)
3245
3246 def get_context_options(self):
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in execute_sql(self, sql, params, commit)
3232 __deprecated__('"commit" has been deprecated and is a no-op.')
3233 logger.debug((sql, params))
-> 3234 with __exception_wrapper__:
3235 cursor = self.cursor()
3236 cursor.execute(sql, params or ())
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in __exit__(self, exc_type, exc_value, traceback)
3008 new_type = self.exceptions[exc_type.__name__]
3009 exc_args = exc_value.args
-> 3010 reraise(new_type, new_type(exc_value, *exc_args), traceback)
3011
3012
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in reraise(tp, value, tb)
190 def reraise(tp, value, tb=None):
191 if value.__traceback__ is not tb:
--> 192 raise value.with_traceback(tb)
193 raise value
194
/tmp/xeus-python-kernel/envs/xeus-python-kernel/lib/python3.10/site-packages/peewee.py in execute_sql(self, sql, params, commit)
3234 with __exception_wrapper__:
3235 cursor = self.cursor()
-> 3236 cursor.execute(sql, params or ())
3237 return cursor
3238
DatabaseError: database disk image is malformed
To reproduce the error, access my JupyterLite Hub and open peewee_test.ipynb at https://michaelweinold.github.io/hub/
According to peewee author Charles Leifer
"the error is not coming from peewee but seems related to however you're abstracting the file-system for use by sqlite. The error is coming from sqlite and seems to indicate some problem with the file."
Could you someone have a look? I know too little about the file system underlying JupyterLite to make any meaningful contributions at this point.
At the moment package simplejson appears to be missing:
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
error libmamba Could not solve for environment specs
The following package could not be installed
└─ simplejson does not exist (perhaps a typo or a missing channel).
add xeus and xeus-kernel to recipes.
Minor problem: all xeus kernels have the same binary name
Working through the Setup Local Build docs, I am encountering issues creating the environment.
Using micromamba to create the environment
micromamba create -n emforge -f ci_env.yml
results in
(base) tylere@tylers-mbp recipes % micromamba create -n emforge -f ci_env.yml
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
conda-forge/osx-arm64 Using cache
conda-forge/noarch Using cache
microsoft/osx-arm64 No change
microsoft/noarch No change
error libmamba Could not solve for environment specs
Encountered problems while solving:
- nothing provides requested patchelf
- nothing provides requested playwright
The environment can't be solved, aborting the operation
critical libmamba Could not solve for environment specs
Similarly, using conda to create the environment
conda env create -n emforge --file ci_env.yml
results in
(base) tylere@tylers-mbp recipes % conda env create -n emforge --file ci_env.yml
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
- playwright
The xeus source is coming from Thorsten's fork
One of the big painpoints wrt python in the browser is the size of packages, in particular the size of scipy is annoying.
But scipy is a collection of a lot of more or less independent submodules(which should have been individual packages in the first place). Some examples of these sub-package are:
scipy.ndimagescipy.sparsescipy.fftscipy.odrscipy.partialMost packages which depend on scipy, for instance pandas, only use a tiny subset of submodules (in the case of pandas it is only uses scipy.sparse).
What about splitting up scipy in a main-package and all the sub-packages.The main-package would then just depend on all sub-packages. If a package depends on scipy it would get the same content as always. But we can change the recipes for packages like pandas st. they only depend on the respective subpackages, ie scipy.sparse.
Unfortunately, I am still not able to reproduce the successful M1/M2 build by @DerThorsten from #277 (comment). Below are my steps, which are pieced together from the documentation and the linked issue.
ci-eny.yml (with playwright installed through pip):name: ci-env
channels:
- conda-forge
dependencies:
- python
- pip
- empack >=2,<3
- colorama
- ruamel
- ruamel.yaml <0.17.6
- rich
- mamba >= 0.23.1
- micromamba >= 0.23.1
- jsonschema
- cython
- patchelf
- liblief
- py-lief
- conda
- conda-build
- pyyaml
- click == 8.0.4
- typer >= 0.7.0
- quetz-client <= 0.0.4
- boa
- nodejs >= 18.7.0
- pydantic
- pytest
- networkx
# - microsoft::playwright
- pip
- pip:
- playwright
conda env create -n emforge --file ci_env.yml
playwright:playwright install
python -m pip install git+https://github.com/DerThorsten/boa.git@python_api --no-deps --ignore-installed
python -m pip install git+https://github.com/emscripten-forge/pyjs-code-runner.git --no-deps --ignore-installed
python -c "from empack.file_packager import download_and_setup_emsdk; download_and_setup_emsdk()"
.emsdkdir file:echo $(python -c "from empack.file_packager import EMSDK_INSTALL_PATH; print(EMSDK_INSTALL_PATH / 'emsdk-3.1.2')") > $HOME/.emsdkdir
python builder.py build explicit recipes/recipes/emscripten_emscripten-32/
python builder.py build explicit recipes/recipes/cross-python_emscripten-32
(emforge) michaelweinold@Michaels-MacBook-Pro emforge-recipes % python builder.py build explicit recipes/recipes_emscripten/regex --emscripten-32
WITH EM
args namespace(recipe_dir=None, target='recipes/recipes_emscripten/regex', features=None, offline=False, target_platform='emscripten-32', json=False, debug=False, variant_config_files=[], interactive=False, output_folder=None, skip_existing='default', no_test=False, continue_on_failure=False, conda_build_build_id_pat=False, conda_build_remove_work_dir=True, conda_build_keep_old_work=False, conda_build_prefix_length=255, croot=False, pkg_format=1, zstd_compression_level=22, post_build_callback=functools.partial(<function post_build_callback at 0x10c7da320>, skip_tests=False, work_dir='/Users/michaelweinold/github/emforge-recipes'), command='build')
Loading config files: /Users/michaelweinold/github/emforge-recipes/conda_build_config.yaml
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.16
WARNING:boa.core.monkeypatch:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.16
Adding in variants from /Users/michaelweinold/github/emforge-recipes/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /Users/michaelweinold/github/emforge-recipes/conda_build_config.yaml
Updating build index: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld
Recipe validation OK
Found 1 recipe
- regex
Assembling all recipes and variants
in build_recipe
Output: regex
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Package ┃ Variant versions ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ target_platform │ emscripten-32 │
│ python │ 3.10.* *_cpython │
│ c_compiler │ emscripten │
│ pip │ 22.0.4 │
└─────────────────┴──────────────────┘
Initializing mamba solver
Downloading source
Source cache directory is: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/src_cache
INFO:conda_build.source:Source cache directory is: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/src_cache
Found source in cache: regex-2022.1.18_97f32dc03a.tar.gz
INFO:conda_build.source:Found source in cache: regex-2022.1.18_97f32dc03a.tar.gz
Extracting download
Preparing environment for regex
Finalizing build for regex
GET SOLVER
https://repo.mamba.pm/emscripten-forge/noarch Using cache
conda-forge/osx-arm64 Using cache
conda-forge/noarch Using cache
pkgs/main/noarch No change
pkgs/main/osx-arm64 No change
pkgs/r/osx-arm64 No change
pkgs/r/noarch No change
https://repo.mamba.pm/emscripten-forge/osx-arm64 94.0 B @ 182.0 B/s 404 failed 0.5s
Reloading output folder: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld
opt/homebrew/Caskroom/miniconda/base/envs/emforg.. 1.2kB @ 26.1MB/s 0.0s
opt/homebrew/Caskroom/miniconda/base/envs/emforg.. 559.0 B @ 24.3MB/s 0.0s
subdir=<libmambapy.bindings.SubdirData object at 0x10ce9a4b0> channel={'platform': 'osx-arm64', 'url': 'file:///opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/osx-arm64', 'channel': opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld[osx-arm64,noarch]}
subdir=<libmambapy.bindings.SubdirData object at 0x10cfafc70> channel={'platform': 'noarch', 'url': 'file:///opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/noarch', 'channel': opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld[osx-arm64,noarch]}
Finalizing host for regex
GET SOLVER
https://repo.mamba.pm/emscripten-forge/emscripten-32 Using cache
https://repo.mamba.pm/emscripten-forge/noarch Using cache
conda-forge/noarch Using cache
pkgs/main/noarch Using cache
pkgs/r/noarch Using cache
pkgs/r/emscripten-32 283.0 B @ 1.4kB/s 404 failed 0.2s
conda-forge/emscripten-32 356.0 B @ 1.7kB/s 404 failed 0.2s
pkgs/main/emscripten-32 285.0 B @ 1.3kB/s 404 failed 0.2s
Reloading output folder: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld
opt/homebrew/Caskroom/miniconda/base/envs/emforg.. ??.?MB @ ??.?MB/s 0 failed 0.0s
opt/homebrew/Caskroom/miniconda/base/envs/emforg.. 559.0 B @ 19.3MB/s 0.0s
subdir=<libmambapy.bindings.SubdirData object at 0x10cfb7770> channel={'platform': 'emscripten-32', 'url': 'file:///opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/emscripten-32', 'channel': opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld[emscripten-32,noarch]}
subdir=<libmambapy.bindings.SubdirData object at 0x10cfb7ef0> channel={'platform': 'noarch', 'url': 'file:///opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/noarch', 'channel': opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld[emscripten-32,noarch]}
Finalizing run for regex
Transaction
Prefix: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/_build_env
Updating specs:
- python=3.10[build=*_cpython]
- cross-python_emscripten-32
- emscripten_emscripten-32
- pip=22.0.4
Package Version Build Channel Size
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Install:
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ bzip2 1.0.8 h3422bc3_4 conda-forge/osx-arm64 Cached
+ ca-certificates 2022.12.7 h4653dfc_0 conda-forge/osx-arm64 Cached
+ coreutils 9.1 h1c322ee_0 conda-forge/osx-arm64 Cached
+ cross-python_emscripten-32 3.10.1 h0edcd7e_8 /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/osx-arm64 Cached
+ crossenv 1.4.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ emscripten_emscripten-32 3.1.27 h0edcd7e_5 /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/osx-arm64 Cached
+ emsdk 3.1.34 py310hbe9552e_0 conda-forge/osx-arm64 Cached
+ gettext 0.21.1 h0186832_0 conda-forge/osx-arm64 Cached
+ icu 70.1 h6b3803e_0 conda-forge/osx-arm64 Cached
+ libcxx 15.0.7 h75e25f2_0 conda-forge/osx-arm64 Cached
+ libffi 3.4.2 h3422bc3_5 conda-forge/osx-arm64 Cached
+ libiconv 1.17 he4db4b2_0 conda-forge/osx-arm64 Cached
+ libsqlite 3.40.0 h76d750c_0 conda-forge/osx-arm64 Cached
+ libuv 1.44.2 he4db4b2_0 conda-forge/osx-arm64 Cached
+ libzlib 1.2.13 h03a7124_4 conda-forge/osx-arm64 Cached
+ lz4-c 1.9.4 hb7217d7_0 conda-forge/osx-arm64 Cached
+ ncurses 6.3 h07bb92c_1 conda-forge/osx-arm64 Cached
+ nodejs 18.15.0 h26a3f6d_0 conda-forge/osx-arm64 Cached
+ openssl 3.1.0 h03a7124_0 conda-forge/osx-arm64 Cached
+ pip 22.0.4 pyhd8ed1ab_0 conda-forge/noarch Cached
+ popt 1.16 h61edf1b_2002 conda-forge/osx-arm64 Cached
+ python 3.10.9 h3ba56d0_0_cpython conda-forge/osx-arm64 Cached
+ python_abi 3.10 3_cp310 conda-forge/osx-arm64 Cached
+ readline 8.1.2 h46ed386_0 conda-forge/osx-arm64 Cached
+ rsync 3.2.7 hcdef6ba_0 conda-forge/osx-arm64 Cached
+ sed 4.8 hc6a1b29_0 conda-forge/osx-arm64 Cached
+ setuptools 59.8.0 py310hbe9552e_1 conda-forge/osx-arm64 Cached
+ tk 8.6.12 he1e0b03_0 conda-forge/osx-arm64 Cached
+ tzdata 2022g h191b570_0 conda-forge/noarch Cached
+ wheel 0.40.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ xxhash 0.8.0 h27ca646_3 conda-forge/osx-arm64 Cached
+ xz 5.2.6 h57fd34a_0 conda-forge/osx-arm64 Cached
+ zlib 1.2.13 h03a7124_4 conda-forge/osx-arm64 Cached
+ zstd 1.5.2 hf913c23_6 conda-forge/osx-arm64 Cached
Summary:
Install: 34 packages
Total download: 0 B
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
Linking libzlib-1.2.13-h03a7124_4
Linking coreutils-9.1-h1c322ee_0
Linking libffi-3.4.2-h3422bc3_5
Linking ncurses-6.3-h07bb92c_1
Linking xz-5.2.6-h57fd34a_0
Linking ca-certificates-2022.12.7-h4653dfc_0
Linking python_abi-3.10-3_cp310
Linking libiconv-1.17-he4db4b2_0
Linking libcxx-15.0.7-h75e25f2_0
Linking libuv-1.44.2-he4db4b2_0
Linking xxhash-0.8.0-h27ca646_3
Linking bzip2-1.0.8-h3422bc3_4
Linking zlib-1.2.13-h03a7124_4
Linking tk-8.6.12-he1e0b03_0
Linking libsqlite-3.40.0-h76d750c_0
Linking readline-8.1.2-h46ed386_0
Linking openssl-3.1.0-h03a7124_0
Linking popt-1.16-h61edf1b_2002
Linking gettext-0.21.1-h0186832_0
Linking zstd-1.5.2-hf913c23_6
Linking lz4-c-1.9.4-hb7217d7_0
Linking icu-70.1-h6b3803e_0
Linking sed-4.8-hc6a1b29_0
Linking rsync-3.2.7-hcdef6ba_0
Linking nodejs-18.15.0-h26a3f6d_0
Linking tzdata-2022g-h191b570_0
Linking python-3.10.9-h3ba56d0_0_cpython
Linking setuptools-59.8.0-py310hbe9552e_1
Linking emsdk-3.1.34-py310hbe9552e_0
Linking wheel-0.40.0-pyhd8ed1ab_0
Linking crossenv-1.4.0-pyhd8ed1ab_0
Linking pip-22.0.4-pyhd8ed1ab_0
Linking emscripten_emscripten-32-3.1.27-h0edcd7e_5
Linking cross-python_emscripten-32-3.10.1-h0edcd7e_8
Transaction finished
Transaction
Prefix: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
Updating specs:
- python=3.10[build=*_cpython]
- python_abi=3.10[build=*_cp310]
- emscripten-abi=3.1.27
Package Version Build Channel Size
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Install:
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ emscripten-abi 3.1.27 h0edcd7e_5 /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/noarch Cached
+ python 3.10.2 h_hash_26_cpython repo.mamba.pm/emscripten-forge/emscripten-32 Cached
+ python_abi 3.10 0_cp310 repo.mamba.pm/emscripten-forge/emscripten-32 Cached
Summary:
Install: 3 packages
Total download: 0 B
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
Linking python_abi-3.10-0_cp310
Linking emscripten-abi-3.1.27-h0edcd7e_5
Linking python-3.10.2-h_hash_26_cpython
Transaction finished
Starting build for regex
source tree in: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work
export
PREFIX=/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
cehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
export BUILD_PREFIX=/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/_build_env
export SRC_DIR=/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work
Found config file /Users/michaelweinold/.emsdkdir
/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/_build_env/etc/conda/activate.d/emscripten_emscripten-32_activate.sh: line 25: /Users/michaelweinold/Library/Caches/empack/emsdk-3.1.2/emsdk_env.sh: No such file or directory
Using EMSCRIPTEN_FORGE_EMSDK_DIR /Users/michaelweinold/.emsdkdir: /Users/michaelweinold/Library/Caches/empack/emsdk-3.1.2
Work directory: /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work
Try building again with /opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work/conda_build.sh
ERROR: Build failed!
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:665 in build │
│ │
│ 662 │ │ │ f.write("\n".join(sorted(list(files_before_script)))) │
│ 663 │ │ │ f.write("\n") │
│ 664 │ │ │
│ ❱ 665 │ │ execute_build_script(m, src_dir, env, provision_only=provision_only) │
│ 666 │ │ │
│ 667 │ │ if provision_only: │
│ 668 │ │ │ return │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:557 in execute_build_script │
│ │
│ 554 │ │ │ │ │ del env["CONDA_BUILD"] │
│ 555 │ │ │ │ │ env["PKG_NAME"] = m.get_value("package/name") │
│ 556 │ │ │ │ │ │
│ ❱ 557 │ │ │ │ │ utils.check_call_env( │
│ 558 │ │ │ │ │ │ cmd, │
│ 559 │ │ │ │ │ │ env=env, │
│ 560 │ │ │ │ │ │ rewrite_stdout_env=rewrite_env, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/conda_build/util │
│ s.py:402 in check_call_env │
│ │
│ 399 │
│ 400 │
│ 401 def check_call_env(popenargs, **kwargs): │
│ ❱ 402 │ return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs) │
│ 403 │
│ 404 │
│ 405 def check_output_env(popenargs, **kwargs): │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/conda_build/util │
│ s.py:382 in _func_defaulting_env_to_os_environ │
│ │
│ 379 │ │ │ out = proc.out.read() │
│ 380 │ │ │
│ 381 │ │ if proc.returncode != 0: │
│ ❱ 382 │ │ │ raise subprocess.CalledProcessError(proc.returncode, _args) │
│ 383 │ │ │
│ 384 │ │ stats.update({'elapsed': proc.elapsed, │
│ 385 │ │ │ │ │ 'disk': proc.disk, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work/conda_build.sh']' returned
non-zero exit status 1.
During handling of the above exception, another exception occurred:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/run_bui │
│ ld.py:376 in build_recipe │
│ │
│ 373 │ │ │ │ f"\n[yellow]Starting build for [bold]{o.name}[/bold][/yellow]\n" │
│ 374 │ │ │ ) │
│ 375 │ │ │ │
│ ❱ 376 │ │ │ final_outputs = build( │
│ 377 │ │ │ │ meta, │
│ 378 │ │ │ │ None, │
│ 379 │ │ │ │ allow_interactive=interactive, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:712 in build │
│ │
│ 709 │ │ │ │ sys.exit() │
│ 710 │ │ else: │
│ 711 │ │ │ console.print("[red]ERROR: Build failed!") │
│ ❱ 712 │ │ │ raise RuntimeError("Build failed") │
│ 713 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Build failed
Build failed
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:665 in build │
│ │
│ 662 │ │ │ f.write("\n".join(sorted(list(files_before_script)))) │
│ 663 │ │ │ f.write("\n") │
│ 664 │ │ │
│ ❱ 665 │ │ execute_build_script(m, src_dir, env, provision_only=provision_only) │
│ 666 │ │ │
│ 667 │ │ if provision_only: │
│ 668 │ │ │ return │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:557 in execute_build_script │
│ │
│ 554 │ │ │ │ │ del env["CONDA_BUILD"] │
│ 555 │ │ │ │ │ env["PKG_NAME"] = m.get_value("package/name") │
│ 556 │ │ │ │ │ │
│ ❱ 557 │ │ │ │ │ utils.check_call_env( │
│ 558 │ │ │ │ │ │ cmd, │
│ 559 │ │ │ │ │ │ env=env, │
│ 560 │ │ │ │ │ │ rewrite_stdout_env=rewrite_env, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/conda_build/util │
│ s.py:402 in check_call_env │
│ │
│ 399 │
│ 400 │
│ 401 def check_call_env(popenargs, **kwargs): │
│ ❱ 402 │ return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs) │
│ 403 │
│ 404 │
│ 405 def check_output_env(popenargs, **kwargs): │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/conda_build/util │
│ s.py:382 in _func_defaulting_env_to_os_environ │
│ │
│ 379 │ │ │ out = proc.out.read() │
│ 380 │ │ │
│ 381 │ │ if proc.returncode != 0: │
│ ❱ 382 │ │ │ raise subprocess.CalledProcessError(proc.returncode, _args) │
│ 383 │ │ │
│ 384 │ │ stats.update({'elapsed': proc.elapsed, │
│ 385 │ │ │ │ │ 'disk': proc.disk, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/opt/homebrew/Caskroom/miniconda/base/envs/emforge/conda-bld/regex-0_1679128758529/work/conda_build.sh']' returned
non-zero exit status 1.
During handling of the above exception, another exception occurred:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/michaelweinold/github/emforge-recipes/builder.py:176 in explicit │
│ │
│ 173 │ if emscripten_32: │
│ 174 │ │ print("WITH EM") │
│ 175 │ │ platform = "emscripten-32" │
│ ❱ 176 │ boa_build( │
│ 177 │ │ work_dir=work_dir, │
│ 178 │ │ target=recipe_dir, │
│ 179 │ │ platform=platform, │
│ │
│ /Users/michaelweinold/github/emforge-recipes/builder.py:132 in boa_build │
│ │
│ 129 │ │ post_build_callback, skip_tests=skip_tests, work_dir=work_dir │
│ 130 │ ) │
│ 131 │ │
│ ❱ 132 │ py_build( │
│ 133 │ │ target=target, │
│ 134 │ │ recipe_dir=recipe_dir, │
│ 135 │ │ target_platform=target_platform, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/pyapi.py:96 │
│ in py_build │
│ │
│ 93 │ │
│ 94 │ from boa.core.run_build import run_build │
│ 95 │ │
│ ❱ 96 │ run_build(args) │
│ 97 │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/run_bui │
│ ld.py:529 in run_build │
│ │
│ 526 │ │ │ │ rerun_build = True │
│ 527 │ │ │ except Exception as e: │
│ 528 │ │ │ │ print(e) │
│ ❱ 529 │ │ │ │ raise e │
│ 530 │ │ │ else: │
│ 531 │ │ │ │ break │
│ 532 │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/run_bui │
│ ld.py:504 in run_build │
│ │
│ 501 │ for recipe in all_recipes: │
│ 502 │ │ while True: │
│ 503 │ │ │ try: │
│ ❱ 504 │ │ │ │ sorted_outputs,final_names = build_recipe( │
│ 505 │ │ │ │ │ args.command, │
│ 506 │ │ │ │ │ recipe["recipe_file"], │
│ 507 │ │ │ │ │ cbc, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/run_bui │
│ ld.py:419 in build_recipe │
│ │
│ 416 │ │ │ │ raise e │
│ 417 │ │ │ else: │
│ 418 │ │ │ │ console.print_exception(show_locals=False) │
│ ❱ 419 │ │ │ │ raise e │
│ 420 │ │
│ 421 │ for o in sorted_outputs: │
│ 422 │ │ if o in failed_outputs: │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/run_bui │
│ ld.py:376 in build_recipe │
│ │
│ 373 │ │ │ │ f"\n[yellow]Starting build for [bold]{o.name}[/bold][/yellow]\n" │
│ 374 │ │ │ ) │
│ 375 │ │ │ │
│ ❱ 376 │ │ │ final_outputs = build( │
│ 377 │ │ │ │ meta, │
│ 378 │ │ │ │ None, │
│ 379 │ │ │ │ allow_interactive=interactive, │
│ │
│ /opt/homebrew/Caskroom/miniconda/base/envs/emforge/lib/python3.10/site-packages/boa/core/build.p │
│ y:712 in build │
│ │
│ 709 │ │ │ │ sys.exit() │
│ 710 │ │ else: │
│ 711 │ │ │ console.print("[red]ERROR: Build failed!") │
│ ❱ 712 │ │ │ raise RuntimeError("Build failed") │
│ 713 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Build failed
The lib\pkgconfig\libxml-2.0.pc files contains the lines:
Libs.private: -L/home/runner/micromamba/envs/ci-env/conda-bld/libxml_1652202972834/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/lib -lm
Cflags: -I${includedir}/libxml2 -I/home/runner/micromamba/envs/ci-env/conda-bld/libxml_1652202972834/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/include
The lib/include and lib/lib should be include and lib instead, and can induce build errors.
Our shared libraries are currently suffixed with *.cpython-310.so. Pyodide uses cpython-310-wasm32-emscripten.so. It's probably a config variable in the sysconfig thing.
Now that the bot is able to open PRs and merge it itself, we should really have some logic to check that:
pip check for Python packages?).Most of the build system seems to be exclusively documented in the github action itself currently:
https://github.com/emscripten-forge/recipes/blob/main/.github/workflows/build_recipes.yaml
It would be nice to provide a script similar to https://github.com/conda-forge/staged-recipes/blob/main/build-locally.py and document the step involved.
At the moment protobuf appears to be missing:
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
error libmamba Could not solve for environment specs
The following package could not be installed
└─ protobuf does not exist (perhaps a typo or a missing channel).
This may resolve #315
consider usage of https://github.com/mymindstorm/setup-emsdk or any pther way to cache the emsdk
For the geos package, we have libgeos and libgeos_c where libgeos_c links dynamically to libgeos. However, in our current configuration, libgeos is not found properly by emscripten.
libgeos.so.3.11.0 is not pre-loaded properlylib directories for the libraries.Latest pytest uses unittest that we removed.
We could either:
We definitely need some more docs, and here are some other things I found more cumbersome than they have to be :)
playwright with pip on M1~/.emsdkdir manually - can we autodetect?builder.py script?~/.condarcemboa file packager fails silently in case of a wrong python version and on other hings
Once it's noarch conda-forge/bokeh-feedstock#76
building embind11 with optimizations gives a runtime error on the JavaScript side complaining about sig being undefined for some javascript funciton.
The related function is en embind::val related function
At the moment the repo shows there is a GitHub Pages environment:
However it gives a 404:
Also there doesn't seem be a gh-pages branch on the repo:
Hi,
Since @DerThorsten provided manual build instructions (thanks!), I tried to reproduce those and ran into a few snags, and also have a few questions, so here goes:
"can be build" -> can be built
aftger -> after
localy -> locally
Bootrap -> Bootstrap
popoulate -> populate
(note: I used my host install of mamba rather than micromamba, is that possible or not (or discouraged)? Some of the issues may be related to that choice) -- maybe the answer would entail using docker rather than attempting a build directly on a developer's machine? My choice was guided by the worry that installing micromamba (which is mainly intended for CI?) alongside regular mamba might trample on the host's mamba install...
ci_env.yaml -> ci_env.yml
conda create -> conda env create
(so the env creation command becomes conda env create -n emforge --file ci_env.yml)
python builder.py build explicit recipes/recipes_emscripten/bzip2 --skip-tests --emscripten-32
.condarc and ran a "full" build (without --skip-tests), which ran until the sanity check (run bzip2 --help) which failed due to node not being found (is there a missing step for activating the emscripten environment?)node works!/home/romain/.cache/empack/emsdk-3.1.2/node/14.18.2_64bit/bin/node /home/romain/mambaforge/envs/emforge/conda-bld/bzip2_1668419287752/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/bzip2.js --help
bzip2, a block-sorting file compressor. Version 1.0.8, 13-Jul-2019.
usage: bzip2.js [flags and input files in any order]
-h --help print this message
-d --decompress force decompression
-z --compress force compression
-k --keep keep (don't delete) input files
-f --force overwrite existing output files
-t --test test compressed file integrity
-c --stdout output to standard out
-q --quiet suppress noncritical error messages
-v --verbose be verbose (a 2nd -v gives more)
-L --license display software version & license
-V --version display software version & license
-s --small use less memory (at most 2500k)
-1 .. -9 set block size to 100k .. 900k
--fast alias for -1
--best alias for -9
If invoked as `bzip2', default action is to compress.
as `bunzip2', default action is to decompress.
as `bzcat', default action is to decompress to stdout.
If no file names are given, bzip2 compresses or decompresses
from standard input to standard output. You can combine
short flags, so `-v -4' means the same as -v4 or -4v, &c.
program exited (with status: 0), but EXIT_RUNTIME is not set, so halting execution but not exiting the runtime or preventing further async execution (build with EXIT_RUNTIME=1, if you want a true shutdown)
But fixing this issue would be nice :-)
I can submit a PR, just let me know the answer to the questions and the relevant / irrelevant parts :-)
To bring the Brightway life-cycle assessment python package to emscripten-forge, we need to add (at the very least) the peewee object-relational mapping (ORM) library (cf. #295) and the lxml XML library.
My understanding is that the package comes with C extensions to provide access to the underlying C libraries, libxml2 and libxslt.
lxml is already included with Pyodide. Compare the pyodide/packages/lxml/meta.yaml file.
Apparently it is possible to use the MuJoCo physics engine with empscripten to do simulation in browser, for example https://kevinzakka.github.io/robopianist-demo/, so it may be cool to have it in emscripten-forge . I do not plan on working on this at the moment, but I opened this issue to point out to anyone interested that a bunch of patches related to this in stillonearth/MuJoCo-WASM#1 (comment) .
Feel free to close, I just wanted to make sure that if somebody searches "mujoco" in this repo he founds this instructions.
Out of our many Brightway conda-forge packages (eg. bw2data, bw2io, bw2calc, etc.) only bw2parameters and bw2speedups are available at beta.mamba.pm/channels/conda-forge:
My understanding is that I can't install packages from the default conda-forge channel during the build process of the xeus-python + JupyterLite hub, instead having to rely on the https://repo.mamba.pm/conda-forge channel.
If this is correct, how can I add our remaining packages?
We should not be needing a jedi recipe as the conda-forge recipe should probably be noarch (see conda-forge/jedi-feedstock#52). Let's remove the jedi recipe from here once it's noarch.
A lot of our ad hoc notebooks could be replaced with browser-based ones, if we could access google bigquery 👍
remove hard coded python version from recipes and build scripts
Looking at the CI on the latest commit, it looks like the HTTP status code: 401 in the "quetz upload packages" CI step gives error, but still succeeds to complete.
For example: https://github.com/emscripten-forge/recipes/actions/runs/4967495499/jobs/8889529799
Run QUETZ_API_KEY=*** quetz-client https://beta.mamba.pm/channels/emscripten-forge ${CONDA_PREFIX}/conda-bld/emscripten-32/*.tar.bz2 || true
Request failed:
HTTP status code: 401
Message: {"detail":"Not logged in"}
Traceback (most recent call last):
File "/home/runner/micromamba-root/envs/ci-env/bin/quetz-client", line 10, in
sys.exit(main())
File "/home/runner/micromamba-root/envs/ci-env/lib/python3.10/site-packages/quetz_client/command_line.py", line 73, in main
files = [('files', open(package, 'rb')) for package in package_file_names]
File "/home/runner/micromamba-root/envs/ci-env/lib/python3.10/site-packages/quetz_client/command_line.py", line 73, in
files = [('files', open(package, 'rb')) for package in package_file_names]
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/micromamba-root/envs/ci-env/conda-bld/linux-64/*.tar.bz2'
Traceback (most recent call last):
File "/home/runner/micromamba-root/envs/ci-env/bin/quetz-client", line 10, in
sys.exit(main())
File "/home/runner/micromamba-root/envs/ci-env/lib/python3.10/site-packages/quetz_client/command_line.py", line 73, in main
files = [('files', open(package, 'rb')) for package in package_file_names]
File "/home/runner/micromamba-root/envs/ci-env/lib/python3.10/site-packages/quetz_client/command_line.py", line 73, in
files = [('files', open(package, 'rb')) for package in package_file_names]
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/micromamba-root/envs/ci-env/conda-bld/noarch/*.tar.bz2'
To bring the Brightway life-cycle assessment python package to emscripten-forge, we need to add (at the very least) the peewee object-relational mapping (ORM) library and the lxml XML library (cf. #311).
My understanding is that the package comes with C extensions
Peewee comes with a couple C extensions that will be built if Cython is available. - peewee setup instructions
more specifically, it is using psycopg2cffi.
Trying to install pyb2d for jupyterlite-xeus-python with the XeusPythonEnv.packages configuration resulted in the following:
with channels:
- https://repo.mamba.pm/emscripten-forge
- https://repo.mamba.pm/conda-forge
The reported errors are:
- Encountered problems while solving:
- - nothing provides requested pyb2d
- - nothing provides pysocks >=1.5.6,<2.0,!=1.5.7 needed by urllib3-1.24.3-py_1
The motivation is to be able to run the Jupyter Games notebooks from https://github.com/jupyterlite/demo/tree/main/content/pyolite/pyb2d, but with jupyterlite-xeus-python.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.