tuxable-ltd/meta-scikit-learn

OpenEmbedded recipes for scikit-learn

meta-scikit-learn's Issues

do_package_write_deb non-existent task

Hi, I am trying to build meta-scikit-learn with the following config:

BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "imx8mqevk"
DISTRO               = "fsl-imx-wayland"
DISTRO_VERSION       = "5.4-zeus"
TUNE_FEATURES        = "aarch64 cortexa53 crc crypto"
TARGET_FPU           = ""
meta                 
meta-poky            = "HEAD:d88d62c20d7d8da85f02edb170dae0280624ad7e"
meta-oe              
meta-multimedia      
meta-python          = "HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-freescale       = "HEAD:14f1a630a47375432f93c556927b879b51d84c4e"
meta-freescale-3rdparty = "HEAD:dbcc686f52c3c84db8cb86aa8973a4e373651b98"
meta-freescale-distro = "HEAD:ca27d12e4964d1336e662bcc60184bbff526c857"
meta-aws             = "HEAD:4dcf78148d951339699b7650319fafda3ce0c3fc"
meta-virtualization  = "zeus:48ec1b49871fef7d331ac1d0680a48d362c7482b"
meta-scikit-learn    = "WIP:39031e8e706bdb57e35d1b85a9fb6f7da3ae61b1"
meta-bsp             
meta-sdk             
meta-ml              = "HEAD:c4620c1b711d180a3f93d69ec6ca416d886f9277"
meta-nxp-demo-experience = "HEAD:c7263d9f3cc7bbf44e7164ffeda494cf283d3dec"
meta-browser         = "HEAD:ee3be3b5986a4aa0e73df2204a625ae1fe5df37e"
meta-rust            = "HEAD:53bfa324891966a2daf5d36dc13d4a43725aebed"
meta-clang           = "HEAD:711e593d5984aad3bf35c51b7ac4482982bc16c7"
meta-gnome           
meta-networking      
meta-filesystems     = "HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-qt5             = "HEAD:21ce4c124d9a972d9122f87c64ac2773bf04c284"
meta-python2         = "HEAD:4400f9155ec193d028208cf0c66aeed2ba2b00ab"

I know this package is supposed to be supported on gatesgarth but I tried zeus just for my purposes. Ran into the following error during compilation:

ERROR: Task do_populate_sdk in /home/ubuntu/environment/imx-yocto-bsp/sources/meta-imx/meta-sdk/dynamic-layers/qt5-layer/recipes-fsl/images/imx-image-full.bb rdepends upon non-existent task do_package_write_deb in /home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scipy-native_1.5.2.bb
ERROR: Command execution failed: 1

Any hints on what to check?

Compatibility with Zeus

Is it possible to use this layer with the Zeus version? Thank you.

Bug in appending python3-scipy?

I have attempted the following zeus build with both meta-scipy and meta-scikit-learn layers included but it failed with the following errors. Renaming python3-scipy_1.3%.bbappend to python3-scipy-native_1.3%.bbappend seemed to fix it. I am not sure what caused the failure in the first place though?

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-18.04"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "imx6ul-var-dart"
DISTRO = "fsl-imx-x11"
DISTRO_VERSION = "5.4-zeus"
TUNE_FEATURES = "arm vfp cortexa7 neon thumb callconvention-hard"
TARGET_FPU = "hard"
meta
meta-poky = "HEAD:0275e68b0576cdefc19718e4b1977044f9f43f48"
meta-oe
meta-multimedia
meta-python = "HEAD:aad5b3d070cd8c58828b0975cf861d8ebc90f460"
meta-freescale = "HEAD:0be39721cea6e04c589a419f4bb07fabd0da6704"
meta-freescale-3rdparty = "HEAD:aea3771baa77e74762358ceb673d407e36637e5f"
meta-freescale-distro = "HEAD:ca27d12e4964d1336e662bcc60184bbff526c857"
meta-bsp
meta-sdk
meta-ml = "HEAD:64918e055f82e7c1d622a28856fb7ca1d1b93b47"
meta-browser = "HEAD:5f365ef0f842ba4651efe88787cf9c63bc8b6cb3"
meta-rust = "HEAD:5cda04c7c2158fc982502ca1e7893f71e4ca1729"
meta-gnome
meta-networking
meta-filesystems = "HEAD:aad5b3d070cd8c58828b0975cf861d8ebc90f460"
meta-qt5 = "HEAD:432ad2aa6c3a13253fefc909faba368851d21fb1"
meta-swupdate = "HEAD:8ad6a75e1778cdda671b3d6aea2a9692613020ec"
meta-virtualization = "HEAD:514ce5f6b240600f06b9956737be1e33900bacac"
meta-variscite-imx = "HEAD:037ae3cab30fcaeb17d465ff41efcb68c0a21a22"
meta-scipy = "zeus:757b20376ed7cb8e4497c6b9bdc5444a5b5a4e81"
meta-scikit-learn = "zeus:d2efaea114efbad76450e130c6c81df50561e565"

Initialising tasks: 100% |#######################################| Time: 0:00:01
Sstate summary: Wanted 8 Found 7 Missed 1 Current 418 (87% match, 99% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: python3-scipy-1.3.3-r0 do_package_qa: QA Issue: /usr/lib/python3.7/site-packages/scipy/optimize/_trlib/_trlib.cpython-37m-arm-linux-gnueabi.so contained in package python3-scipy requires libblas.so.3, but no providers found in RDEPENDS_python3-scipy? [file-rdeps]
ERROR: python3-scipy-1.3.3-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /home/jhuang/var-fsl-yocto/build_x11/tmp/work/cortexa7t2hf-neon-poky-linux-gnueabi/python3-scipy/1.3.3-r0/temp/log.do_package_qa.4163
ERROR: Task (/home/jhuang/var-fsl-yocto/sources/meta-scipy/recipes-devtools/python/python3-scipy_1.3.3.bb:do_package_qa) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1696 tasks of which 1695 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/home/jhuang/var-fsl-yocto/sources/meta-scipy/recipes-devtools/python/python3-scipy_1.3.3.bb:do_package_qa
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Lack of license

I'm confused under what license this layer is distributed, would be happy to see LICENSE file as in other your repos

QA issue python3-scipy requires libblas

I'm using dunfell and when bitbaking I get an QA issue

ERROR: python3-scipy-1.3.3-r0 do_package_qa: QA Issue: /usr/lib/python3.8/site-packages/scipy/integrate/vode.cpython-38-aarch64-linux-gnu.so contained in package python3-scipy requires libblas.so.3()(64bit), but no providers found in RDEPENDS_python3-scipy? [file-rdeps]
ERROR: python3-scipy-1.3.3-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.

When I'm not including meta-scikit-learn bitbake python3-scipy is running without error.

Does anyone know how to proceed here?

32bit arm failure

Some function of scikit-learn fail on an arm 32 bit system. To reproduce build a scitkit-learn enabled image for arm, an example conf looks like:

Build Configuration:
BB_VERSION           = "1.49.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "arch"
TARGET_SYS           = "arm-oe-linux-gnueabi"
MACHINE              = "qemuarm"
DISTRO               = "nodistro"
DISTRO_VERSION       = "nodistro.0"
TUNE_FEATURES        = "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU           = "hard"
meta                 = "master:68f20ac5524a5db8a09a90df796b4862cf747d3c"
meta-oe              
meta-python          = "master:759058bfb832161eb0d21f5c51a49c8361e727bc"
meta-scikit-learn    = "<unknown>:<unknown>"

local.conf additions

MACHINE ?= "qemuarm"

IMAGE_INSTALL_append = "python3-scikit-learn"
EXTRA_IMAGE_FEATURES_append = " ssh-server-dropbear"

then

bitbake core-image-minimal

then

runqemu

once booted you can SSH into the qemu with

ssh [email protected]

then run the pytest for the dbscan_inner

pytest /usr/lib/python3.9/site-packages/sklearn/cluster/tests/test_dbscan.py

which will output the following error

================================================ test session starts =================================================
platform linux -- Python 3.9.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /usr/lib/python3.9/site-packages/sklearn/cluster
collected 28 items                                                                                                   

../../usr/lib/python3.9/site-packages/sklearn/cluster/tests/test_dbscan.py .FFFFatal Python error: Segmentation fault

Current thread 0xb6f76010 (most recent call first):
  File "/usr/lib/python3.9/site-packages/sklearn/neighbors/_base.py", line 503 in _fit
  File "/usr/lib/python3.9/site-packages/sklearn/neighbors/_unsupervised.py", line 142 in fit
  File "/usr/lib/python3.9/site-packages/sklearn/cluster/tests/test_dbscan.py", line 85 in test_dbscan_sparse_precomputed
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 1641 in runtest
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 311 in from_call
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 323 in _main
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/usr/bin/pytest", line 33 in <module>

I have identified this as a 64bit -> 32bit compile issue, probably from using the -native include files when building. The exact error from our proprietary codebase looks like:

File "sklearn/cluster/_dbscan_inner.pyx", line 40, in sklearn.cluster._dbscan_inner.dbscan_inner
ValueError: Buffer dtype mismatch, expected 'npy_intp' but got 'long long'

Which I think makes it obvious that it's using a long long type as an integer pointer which would be valid on an 64bit system, but not on a 32bit system.