Comments (8)
Thanks for reporting! @gauravsaini04 can you prioritize investigating the issue? thanks!
from images.
FYI, I made a smaller reproduction here:
https://github.com/bpkroth/miniconda-devcontainer-test
I think it might actually have something to do with pandas being installed from conda and pyarrow being installed from pip.
In the original example this was unintentional, but here I did it explicitly.
Anyways, still unclear why the base devcontainer change should affect that.
from images.
Thanks for reporting! @gauravsaini04 can you prioritize investigating the issue? thanks!
sure @samruddhikhandale, searching for cause of the issue
from images.
Thanks @bpkroth for providing us more information. The repro repo is pretty helpful, conda env update -n base -v -f environment.yml
command definitely fails with the latest image
I think it might actually have something to do with pandas being installed from conda and pyarrow being installed from pip.
The only change that went into latest version of the miniconda image is #976 which did something similar to ^ ; where cryptography
installation was moved from conda to pip. I am not confident if that's the root cause, however, @gauravsaini04 I can think of few ways to debug the root cause -
- Can you use https://github.com/bpkroth/miniconda-devcontainer-test for repro, and instead of using the base image, can you build the miniconda image with reverted #976 changes?
- Can you check if there was a newer release of continuumio/miniconda3 which could have caused this?
from images.
@prathameshzarkar9, had been working on this issue.
He has been testing the issue with the following scenarios, as suggested by you :
Scenario #1 : Test with the following environment.yml
in the test project by @bpkroth,
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pyarrow
- Sub-scenario #1 : Test with the old miniconda upstream image i.e. in 0.203.6-3
- Sub-scenario #2 : Test with the latest miniconda upstream image i.e. 0.203.7
Scenario #2 : Test with the following environment.yml
,
channels:
- defaults
dependencies:
- pip:
- python
- pandas
- pytest
- pip
- pyarrow
- Sub-scenario #1 : Testing with the old miniconda image
- Sub-scenario #2 : Testing with the new miniconda image
Scenario #3 : Testing the default environment.yml
i.e.
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pip:
- pyarrow
- Sub-scenario #1 : Testing with old miniconda image
- Sub-scenario #2 : Testing with new miniconda image
if all these scenarios fail with the similar error message, then it implies that the issue is obviously related to the upstream miniconda
image (released just after the 0.203.6-3
i.e. the image present in our 0.203.7
image).
from images.
Hi @gauravsaini04 @samruddhikhandale ,
I have tested all the scenarios mentioned above.
Scenario 1 - sub scenario 1:
using mcr.microsoft.com/devcontainers/miniconda:0.203.6-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
conda run -n base pytest test_exec_example.py
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/conda-tmp
collected 1 item
test_exec_example.py . [100%]
============================== 1 passed in 0.38s ===============================
Scenario 1 - sub scenario 2:
using mcr.microsoft.com/devcontainers/miniconda:0.203.7-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
conda run -n base pytest test_exec_example.py
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/conda-tmp
collected 1 item
test_exec_example.py . [100%]
============================== 1 passed in 0.39s ===============================
Scenario 2 - sub scenario 1:
using mcr.microsoft.com/devcontainers/miniconda:0.203.6-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- pip:
- python
- pandas
- pytest
- pip
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
conda env update -n base -v -f environment.yml
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/conda/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/conda-tmp/condaenv.ht5xghx7.requirements.txt', '--exists-action=b']
Pip subprocess output:
Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement python (from versions: none)
ERROR: No matching distribution found for python
failed
CondaEnvException: Pip failed
Scenario 2 - sub scenario 2:
using mcr.microsoft.com/devcontainers/miniconda:0.203.7-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- pip:
- python
- pandas
- pytest
- pip
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
conda env update -n base -v -f environment.yml
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/conda/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/conda-tmp/condaenv.ht5xghx7.requirements.txt', '--exists-action=b']
Pip subprocess output:
Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement python (from versions: none)
ERROR: No matching distribution found for python
failed
CondaEnvException: Pip failed
Scenario 3 - sub scenario 1:
using mcr.microsoft.com/devcontainers/miniconda:0.203.6-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pip:
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
done
To activate this environment, use
conda activate base
To deactivate an active environment, use
conda deactivate
conda run -n base pytest test_exec_example.py
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/conda-tmp
collected 1 item
test_exec_example.py F [100%]
=================================== FAILURES ===================================
______________________________ test_exec_example _______________________________
def test_exec_example():
env = {}
env["PATH"] = os.environ["PATH"]
env["foo"] = "bar"
result = run(["./example.py"],
capture_output=True,
text=True,
check=False,
env=env,
cwd=os.path.realpath(os.path.dirname(__file__))
)
result.check_returncode()
test_exec_example.py:15:
self = CompletedProcess(args=['./example.py'], returncode=1, stdout='', stderr='Traceback (most recent call last):\n File "/...quired by /opt/conda/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)\n')
def check_returncode(self):
"""Raise CalledProcessError if the exit code is non-zero."""
if self.returncode:
raise CalledProcessError(self.returncode, self.args, self.stdout,
self.stderr)
E subprocess.CalledProcessError: Command '['./example.py']' returned non-zero exit status 1.
/opt/conda/lib/python3.11/subprocess.py:502: CalledProcessError
=========================== short test summary info ============================
FAILED test_exec_example.py::test_exec_example - subprocess.CalledProcessErro...
============================== 1 failed in 0.32s ===============================
ERROR conda.cli.main_run:execute(125): conda run pytest test_exec_example.py
failed. (See above for error)
Scenario 3 - sub scenario 2:
using mcr.microsoft.com/devcontainers/miniconda:0.203.7-3 to build miniconda test project with following environment.yml
channels:
- defaults
dependencies:
- python
- pandas
- pytest
- pip
- pip:
- pyarrow
Set the variable:
LD_LIBRARY_PATH=/opt/conda/lib
Executed:
test.sh
Output:
done
To activate this environment, use
conda activate base
To deactivate an active environment, use
conda deactivate
conda run -n base pytest test_exec_example.py
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/conda-tmp
collected 1 item
test_exec_example.py F [100%]
=================================== FAILURES ===================================
______________________________ test_exec_example _______________________________
def test_exec_example():
env = {}
env["PATH"] = os.environ["PATH"]
env["foo"] = "bar"
result = run(["./example.py"],
capture_output=True,
text=True,
check=False,
env=env,
cwd=os.path.realpath(os.path.dirname(__file__))
)
result.check_returncode()
test_exec_example.py:15:
self = CompletedProcess(args=['./example.py'], returncode=1, stdout='', stderr='Traceback (most recent call last):\n File "/...quired by /opt/conda/lib/python3.12/site-packages/pandas/_libs/window/aggregations.cpython-312-x86_64-linux-gnu.so)\n')
def check_returncode(self):
"""Raise CalledProcessError if the exit code is non-zero."""
if self.returncode:
raise CalledProcessError(self.returncode, self.args, self.stdout,
self.stderr)
E subprocess.CalledProcessError: Command '['./example.py']' returned non-zero exit status 1.
/opt/conda/lib/python3.12/subprocess.py:502: CalledProcessError
=========================== short test summary info ============================
FAILED test_exec_example.py::test_exec_example - subprocess.CalledProcessErro...
============================== 1 failed in 0.34s ===============================
ERROR conda.cli.main_run:execute(125): conda run pytest test_exec_example.py
failed. (See above for error)
Hence, from the above tests, we can conclude below 2 things:
- we need to set the environment variable in miniconda project dockerfile as : LD_LIBRARY_PATH=/opt/conda/lib
- whoever is using this image, they have to make sure that the environment.yml file should contain all the packages under conda channel itself and not under the pip channel
Please find the PR created for these changes:
PR #1075
from images.
Hi @samruddhikhandale ,
It is important to note that setting LD_LIBRARY_PATH can have unintended consequences, as it can cause the dynamic linker to find and load a different version of a library than the one the program was built with. This can potentially lead to unexpected behavior or crashes. Therefore, it is generally recommended to avoid using LD_LIBRARY_PATH if possible, and to instead install any required libraries in the standard system directories.
Additionally, it is generally better to set the LD_LIBRARY_PATH variable in a per-user or per-application basis, rather than modifying the system-wide /etc/environment file. This can be done by adding the export line to the user's shell initialization file, such as ~/.bashrc or ~/.bash_profile.
Best Practices
To use LD_LIBRARY_PATH effectively and safely:
- Minimize Scope: Set LD_LIBRARY_PATH only in the environment of the specific application that needs it, not globally.
- Wrapper Scripts: Use wrapper scripts to set LD_LIBRARY_PATH for applications, ensuring other parts of the system remain unaffected.
- Temporary Use: Use it temporarily during development, testing, or troubleshooting, and remove it for production deployments.
- System Configuration: For long-term solutions, prefer modifying system-wide library configurations using ld.so.conf and running ldconfig.
What should not set this variable globally as it is not recommended. This variable should be handled as per user requirements in their specific user context.
from images.
Thank you @prathameshzarkar9 for the detailed summary and investigation. Great findings 👏
Let’s close #1075
Closing this issue as we won’t be able to fix this on our end, and directly setting LD_LIBRARY_PATH
in the image is risky as stated in #989 (comment).
whoever is using this image, they have to make sure that the environment.yml file should contain all the packages under conda channel itself and not under the pip channel
@bpkroth Can you continue using LD_LIBRARY_PATH
in your dev config or ensure you follow ^ to fix this issue?
Let's reopen this issue if needed, thank you!
from images.
Related Issues (20)
- Support Ubuntu 24.04 scheduled for April 25th HOT 4
- [DERIK GORDON] [ ] EOL scheduled for [January 31, 1969]/[April 03, 2024] HOT 1
- Can't update Rust in bookworm? HOT 4
- Support Cpp (Ubuntu-24.04) HOT 3
- [Image Name] [variant/version] EOL scheduled for [date] HOT 2
- [Image Name] [variant/version] EOL scheduled for [date]
- vcpkg missing from cpp images
- Support Alpine 3.20 scheduled for May 2024 HOT 3
- add opencontainers labels to the docker images HOT 5
- uid mapping problem with ubuntu-24.04 base image HOT 5
- [Universal] Unpin patch version for dotnet runtime & core HOT 1
- [variant/version] EOL scheduled for [date]
- samruddhikhandale/cpp-ts HOT 3
- Is DevContainer Support distroless python docker images? HOT 2
- Python standard library Modulefinder not working with editable installed packages HOT 1
- SSL certificate problem: unable to get local issuer certificate HOT 2
- Alpine 3.16 EOL HOT 2
- > @prathameshzarkarollowing Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information. HOT 1
- Gem subdirectory not writable on ruby:3.2-bookworm
- [Image Name] [variant/version] EOL scheduled for [date] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from images.