thoth-station/ps-cv

This is a repository for a Predictive Stack for Computer Vision (CV)

Predictable Stack for Computer Vision Applications

The images described here are built with Thoth Predictable Stacks and contains Computer Vision packages for Data Science usage.

List of packages in ps-cv-ocr

- opencv-python
- pytesseract
- pillow

This image contain specific binaries for openv-cv and tesseract. In particular tesseract languages added are Finnish, Swedish and English).

List of packages in ps-cv-tensorflow

- opencv-python
- tensorflow
- pillow
- keras
- tensorboard
- tensorflow
- jupyter-tensorboard

This image contain specific binaries for openv-cv.

List of packages in ps-cv-pytorch

- opencv-python
- torch
- torchvision
- pillow
- tensorboard
- jupyter-tensorboard

This image contain specific binaries for openv-cv.

Generic data science packages

All the above images are built from Generic Data Science Image, therefore they contain also the following packages:

- beautifulsoup4
- bokeh
- boto3
- cloudpickle
- cython
- dask
- dill
- distributed
- h5py
- ipywidgets
- kafka-python
- matplotlib
- numpy
- pandas
- plotly
- pyarrow
- s3fs
- scikit-image
- scikit-learn
- scipy
- seaborn
- sqlalchemy
- statsmodels

ps-cv's Issues

Issue during overlays build image

Describe the bug
Issue during overlays build image of ps-cv-tensorflow

To Reproduce
Steps to reproduce the behavior:

1. start tag release pipeline in this repo

Expected behavior
Image is built correctly

Screenshots

error building at STEP "ENABLE_MICROPIPENV="1" ": Build error: Unknown instruction: "ENABLE_MICROPIPENV=\"1\"" &imagebuilder.Step{Env:[]string{"PATH=/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=oci", "SUMMARY=Thoth's Source-to-Image for Python 3.8 applications", "DESCRIPTION=Thoth's Source-to-Image for Python 3.8 applications. This toolchain is based on Red Hat UBI8. It includes pipenv.", "STI_SCRIPTS_URL=image:///usr/libexec/s2i", "STI_SCRIPTS_PATH=/usr/libexec/s2i", "APP_ROOT=/opt/app-root", "HOME=/opt/app-root/src", "PLATFORM=el8", "NODEJS_VER=10", "PYTHON_VERSION=3.8", "PYTHONUNBUFFERED=1", "PYTHONIOENCODING=UTF-8", "LC_ALL=en_US.UTF-8", "LANG=en_US.UTF-8", "PIP_NO_CACHE_DIR=off", "BASH_ENV=/opt/app-root/etc/scl_enable", "ENV=/opt/app-root/etc/scl_enable", "PROMPT_COMMAND=. /opt/app-root/etc/scl_enable", "THOTH_S2I_NAME=quay.io/thoth-station/s2i-thoth-ubi8-py38", "THOTH_S2I_VERSION=0.26.0", "THAMOS_NO_PROGRESSBAR=1", "THAMOS_NO_EMOJI=1", "JUPYTER_LAB_ENABLE=1", "ENABLE_MICROPIPENV=1", "JUPYTER_ENABLE_LAB=true", "UPGRADE_PIP_TO_LATEST=1", "IMAGE_NAME=quay.io/thoth-station/s2i-generic-data-science-notebook", "IMAGE_TAG=v0.0.4", "THAMOS_RUNTIME_ENVIRONMENT=ps-cv-tensorflow", "THOTH_ADVISE=0", "THOTH_ERROR_FALLBACK=1", "THOTH_DRY_RUN=1", "THAMOS_DEBUG=0", "THAMOS_VERBOSE=1", "THOTH_PROVENANCE_CHECK=0", "GIT_REPO_NAME=s2i-generic-data-science-notebook"}, Command:"enable_micropipenv=\"1\"", Args:[]string{""}, Flags:[]string{}, Attrs:map[string]bool(nil), Message:"ENABLE_MICROPIPENV=\"1\" ", Original:"ENABLE_MICROPIPENV=\"1\"     UPGRADE_PIP_TO_LATEST=\"1\""}

Additional context

New minor release

Hey, Kebechet!

Create a new minor release, please.

"

ps-cv/pyproject.toml

Lines 2 to 7 in 6626476

	name = "TODO"
	description = "TODO"
	readme = "README.rst"
	requires-python = ">=3.8"
	license = {file = "LICENSE"}

---

This issue was generated by todo based on a TODO comment in 6626476. It's been assigned to @goern because they committed the code.

Missing Pipfile

Check your repository to make sure Pipfile exists

Kebechet update

Hey, Kebechet!

Update my dependencies, please.

Tensorflow in ps-cv-tensorflow doesn't recognize xla_gpu

Describe the bug
Tensorflow (version '2.6.0-rc0') doesn't recognize xla_gpu (Tesla v100 in this case). Running nvidia-smi gives me: "NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2".

To Reproduce
Steps to reproduce the behavior:
import tensorflow as tf
tf.test.is_gpu_available() #This gives me False
tf.config.list_physical_devices() #This returns [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')] for me

New patch release

Hey, Kebechet!

Create a new patch release, please.

New patch release

Hey, Kebechet!

Create a new patch release, please.

Podman build of overlay image fails at COPY .s2i/bin /tmp/scripts

Describe the bug
Attempting to build an overlay image is failing on my local workstation

To Reproduce
Steps to reproduce the behavior:

1. git clone https://github.com/thoth-station/ps-cv
2. cd ps-cv
3. podman build overlays/ps-cv-pytorch
4. See error

Error: error building at STEP "COPY .s2i/bin /tmp/scripts": checking on sources under "/home/jason/aicoe/github/ps-cv/overlays/ps-cv-pytorch": copier: stat: "/.s2i/bin": no such file or directory

Expected behavior
Successful build of the ps-cv-pytorch image

Screenshots

Additional context

ps-cv-pytorch -- pillow 8.3.0 has a bug

Describe the bug
Pytorch doesn't work correctly with pillow 8.3.0 (https://stackoverflow.com/questions/68223871/pytorch-cifar10-dataset-cannot-get-first-item). Downgrading pillow to 8.2.0 seems to solve the issue.

To Reproduce
Steps to reproduce the behavior:

1. train_data = torchvision.datasets.ImageFolder(root="data/...", transform=TRANSFORM_IMG).

Kebechet Advise

Testing/verifying that the advise manager works on this repo

New patch release

Hey, Kebechet!

Create a new patch release, please.

IMPORTANT NOTES

• If Khebut GitHub App Bot is installed, this issue will trigger a patch release. The bot will open a Pull Request to update the CHANGELOG, fix the opened issue and create a tag.

• Only users that are allowed to release (a.k.a. maintainers specified in the .thoth.yaml file) can open the issue, otherwise bot will reject them, commenting and closing the issue. If AICoE CI GitHub App is installed, once the pull request is merged and a new tag is created by the bot, the pipeline to build and push image starts.

Kebechet Advise

Hey, Kebechet!

Please update my dependencies using advice from Thoth.

Let's see if your current release works better than #32

No dependency management found

No dependency management found for this repository. If you want to keep your dependencies managed, please submit Pipfile or requirements.in or requirements-dev.in file.

To generate a Pipfile, use:

$ pipenv install --skip-lock --code ./
$ git add Pipfile
$ git commit -m 'Add Pipfile for dependency management'

Make sure your Pipfile or requirements.in or requirements-dev.in is placed in the root of your Git repository.

Create computer vision image for OCR, image analysis and image classification with deep learning

Is your feature request related to a problem? Please describe.
As Data Scientist working on computer visions projects,

I want to have an image with some specific libraries for my computer vision projects.

Describe the solution you'd like

• choose naming convention
• Set repo for computer vision image with the following libraries:

OpenCV

·        Tesseract 4. For Tesseract it would be useful to have OCR models for many different languages included in the image so that the image would be useful around the world. We’d need OCR models for Finnish, Swedish and English.

·        Python libraries:

o   Tensorflow

o   Keras

o   pytesseract

o   pillow

o   numpy

o   scikit-image

o   pandas

o   scikit-learn

o   matplotlib

• have a test for the image

Describe alternatives you've considered

Additional context

cc @harshad16

No dependency management found

No dependency management found for this repository. If you want to keep your dependencies managed, please submit Pipfile or requirements.in or requirements-dev.in file.

To generate a Pipfile, use:

$ pipenv install --skip-lock --code ./
$ git add Pipfile
$ git commit -m 'Add Pipfile for dependency management'

Make sure your Pipfile or requirements.in or requirements-dev.in is placed in the root of your Git repository.

Failed to update dependencies to their latest version for ps-cv-tensorflow environment

Automatic dependency update failed for the current master with SHA 61d2513.

The automatic dependency management cannot continue. Please fix errors reported bellow.

Command

  $ pipenv update --dev

Standard output

Running $ pipenv lock then $ pipenv sync.
Installing dependencies from Pipfile.lock (218bea)...
Installing initially failed dependencies...

Standard error

Creating a virtualenv for this project...
Pipfile: /tmp/clone-dir/overlays/ps-cv-tensorflow/Pipfile
Using /home/user/.local/share/virtualenvs/user-5PivF_Od/bin/python3.8 (3.8.12) to create virtualenv...
created virtual environment CPython3.8.12.final.0-64 in 914ms
  creator CPython3Posix(dest=/tmp/clone-dir/overlays/ps-cv-tensorflow/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/tmp/tmpq8xz05iy)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator


Successfully created virtual environment!
�[KVirtualenv location: /tmp/clone-dir/overlays/ps-cv-tensorflow/.venv
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Building requirements...
�[K
Resolving dependencies...
�[K
Success!
�[KUpdated Pipfile.lock (218bea)!
An error occurred while installing tensorflow==2.9.1 --hash=sha256:adc1b80ec32337ba1720fadbf64242aae06a7ee1c7aea6a1a0cffe1968c89b25 --hash=sha256:add2f4e55fd7209a5cb01d815364adc9827184d74a83c2f1616c72be2e85c8b7 --hash=sha256:488a7a964298b92278cca593b7f687abd1c8f51982a08dc6ded5efd9457bf477 --hash=sha256:abbdcfa7c14974f9bafa8f51214c037639a8991eef3dbcafc506e608a673c54c --hash=sha256:8be7d7f6091ac5af09c19eb178bbbd51346794684851a7bd57826d2d1cfa220e --hash=sha256:b1f7674220b19ab5d39d9313f9c60278cef56052614600cba192700c658e502c --hash=sha256:81ad045d39cf4360d5e6183cdda0990661302cca502917a12e54177989b340b9 --hash=sha256:df048d216f90c9fcbda22bdd03420b0e0e6e7ee98b8b91f221afef46b7664635 --hash=sha256:212a4d97fb6dedff257703d7a2ce50b0b8ec23d955938083363aa7bc526918cc --hash=sha256:1c2c6557f6bd7e0d3f1063e27346e4f1c4353db2b7751ac7451033b335fa4b7c --hash=sha256:a27bffb91f0aef6ba2a30f6507932d9c6801af56cfee8442767f9f3f69bb37a6 --hash=sha256:2c77edeb4a9d542032ddac93f732e3dd3d1c4d15c8795e4e43166ed46503582d! Will try again.
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/cli/command.py", line 594, in update
[InstallError]:       do_sync(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 2884, in do_sync
[InstallError]:       do_init(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]:       do_install_dependencies(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]:       batch_install(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Collecting tensorflow==2.9.1
ERROR: Couldn't install package: tensorflow
 Package installation failed...

Environment details

Kebechet version: 1.10.3
Python version: 3.8.12
Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.2.5
pipenv version: pipenv, version 2020.11.15

Dependency graph

Unable to obtain dependency graph:

An error occurred while installing tensorflow==2.9.1 --hash=sha256:2c77edeb4a9d542032ddac93f732e3dd3d1c4d15c8795e4e43166ed46503582d --hash=sha256:add2f4e55fd7209a5cb01d815364adc9827184d74a83c2f1616c72be2e85c8b7 --hash=sha256:81ad045d39cf4360d5e6183cdda0990661302cca502917a12e54177989b340b9 --hash=sha256:8be7d7f6091ac5af09c19eb178bbbd51346794684851a7bd57826d2d1cfa220e --hash=sha256:a27bffb91f0aef6ba2a30f6507932d9c6801af56cfee8442767f9f3f69bb37a6 --hash=sha256:212a4d97fb6dedff257703d7a2ce50b0b8ec23d955938083363aa7bc526918cc --hash=sha256:df048d216f90c9fcbda22bdd03420b0e0e6e7ee98b8b91f221afef46b7664635 --hash=sha256:488a7a964298b92278cca593b7f687abd1c8f51982a08dc6ded5efd9457bf477 --hash=sha256:b1f7674220b19ab5d39d9313f9c60278cef56052614600cba192700c658e502c --hash=sha256:1c2c6557f6bd7e0d3f1063e27346e4f1c4353db2b7751ac7451033b335fa4b7c --hash=sha256:adc1b80ec32337ba1720fadbf64242aae06a7ee1c7aea6a1a0cffe1968c89b25 --hash=sha256:abbdcfa7c14974f9bafa8f51214c037639a8991eef3dbcafc506e608a673c54c! Will try again.
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/cli/command.py", line 684, in sync
[InstallError]:       retcode = do_sync(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 2884, in do_sync
[InstallError]:       do_init(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]:       do_install_dependencies(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]:       batch_install(
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.8/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Collecting tensorflow==2.9.1
ERROR: Couldn't install package: tensorflow
 Package installation failed...

Notes

For more information, see Pipfile and Pipfile.lock.

Once this issue is resolved, the issue will be automatically closed by bot.

/label thoth/potential-observation
/kind bug
/priority critical-urgent