Latest build doesn't work on Linux

[trelau]

Latest build doesn't work on Linux

[looooo]

let's see: conda-forge/smesh-feedstock#8

[trelau]

@shrdluk @looooo Linux builds with SMESH and Netgen should be available on my trelau Anaconda channel if you want to try them out (tests were successful on Travis and Appveyor). I am working on a conda-forge recipe, may generate a beta release of pyOCCT for that.

[trelau]

I don't have an __init__.py file in the test folder, maybe that is causing the error when trying to automatically run the unittests? Have you tried python test_graphic3d.py from a prompt inside the test folder? I should be running these tests during the build...

[shrdluk]

I don't think it's the lack of an init.py file. The tests ran OK on the previous version, and as you'll see from the above session, I cd-ed into the test/ folder before running the tests.

From the undefined symbol error, I wonder if the project is being built against a different OCCT version from the intended one. I seem to remember @looooo seeing similar errors when he had a wrong version in his build.

I've seen the 'pybind11::error_already_set' message when a module which is referred to by other modules isn't present. It's a pity the error message isn't more helpful than "you've made so many errors I can't even begin to tell you where you've gone wrong" 😄

Presumably the project builds and tests OK on Windows, so it can't be anything too fundamental. Perhaps the versions are somehow different between Windows and Linux.

[trelau]

Ah I didn't see the undefined symbol error, just saw the ImportError on my phone...

The symbol THE_DEFAULT_DEPTH is a const static attribute, maybe for whatever reason that doesn't get exposed in Linux vs Windows OCCT builds? I could try commenting these two lines https://github.com/LaughlinResearch/pyOCCT/blob/master/src/Graphic3d_1.cpp#L914-L915?

[shrdluk]

I think that's probably it. I've just been reading about symbol visibility in gcc, and it appears that the situation is different between Linux and Windows. I tried looking for THE_DEFAULT_DEPTH and other static attribute symbols in the OCCT .so files, and failed to find them:

$ nm -DC lib* | grep THE_
$

and

$ nm -DC lib* | grep TransparencyMethod
$

So commenting out the two lines might just means it fails on the next static attribute symbol.

Visibility is determined at compile / link time, and I guess the conda-forge OCCT for Linux has been built with visibility hidden for these symbols. Perhaps @looooo can comment on this.

Following up your earlier comment, it would be good to run the tests as part of the build. I guess this could be done in a separate job, so it wouldn't eat into the build time.

[trelau]

hmm...ok well i'll first run the tests during the build and see what happens...if the only thing that fails on Linux is the test of the static attributes, then i'll just comment them out for now

edit: maybe something relevant here? https://pybind11.readthedocs.io/en/stable/faq.html#someclass-declared-with-greater-visibility-than-the-type-of-its-field-someclass-member-wattributes

[shrdluk]

The pybind11 item is interesting, but not quite the same thing. They are saying that having -fvisibility=hidden is necessary for pybind11, and a good thing generally, since it reduces code size etc, especially with C++ and heavy use of templates. The symbol visibility article above says much the same thing.

Our issue is that the symbols we want, the static attributes, have apparently been made hidden in OCCT, and we need them unhidden. We don't want a general departure from the -fvisibility=hidden policy, just an exemption for a few symbols.

I imagine this will also affect C++ programs that use the static attributes and link to the OCCT Linux build.

[looooo]

Visibility is determined at compile / link time, and I guess the conda-forge OCCT for Linux has been built with visibility hidden for these symbols. Perhaps @looooo can comment on this.

@shrdluk https://github.com/conda-forge/occt-feedstock/blob/master/recipe/build.sh
I didn't set any special fflags for occt. Maybe it is set by default? Also I never tested any visual moduls of occt.

[trelau]

@shrdluk I'm not sure the latest build actually ran the test on linux...could you have a look at the Travis-CI results towards the bottom and test out the conda packages? I removed the static const attributes too. The Appveyor builds actually failed for some reason...problem for another night...

[shrdluk]

@shrdluk https://github.com/conda-forge/occt-feedstock/blob/master/recipe/build.sh
I didn't set any special fflags for occt. Maybe it is set by default? Also I never tested any visual moduls of occt.

Looking at your conda recipe, I agree. There's nothing there that would affect visibility.

I also ran grep on the OCCT source and failed to find any occurrences of -fvisibility or __attribute__ ((visibility ..., so I conclude that OCCT is using the default (non-hidden) visibility settings.

Thinking about it further, I've decided that visibility is a red herring. The Graphic3d_RenderingParams data fields and static attributes are internal and shouldn't have loader symbols associated with them.

I think what may be happening is that with gcc, the static public attributes THE_DEFAULT_RESOLUTION and THE_DEFAULT_DEPTH aren't being treated as fields of Graphic3d_RenderingParams, but as external constants to be resolved by the loader at run time. Since there are no such symbols, we get the undefined symbol error.

It will be interesting to see if the data fields are also being treated in this way.

Why Visual Studio and gcc handle this differently is another question.

[shrdluk]

@shrdluk I'm not sure the latest build actually ran the test on linux...could you have a look at the Travis-CI results towards the bottom and test out the conda packages?

From the Travis CI log:

TEST START: /home/travis/miniconda/conda-bld/linux-64/pyocct-7.2.0.1-py36_0.tar.bz2
Updating index at /home/travis/miniconda/conda-bld/linux-64 to make package installable with dependencies
INFO:conda_build.build:Updating index at /home/travis/miniconda/conda-bld/linux-64 to make package installable with dependencies
Updating index at /home/travis/miniconda/conda-bld/noarch to make package installable with dependencies
INFO:conda_build.build:Updating index at /home/travis/miniconda/conda-bld/noarch to make package installable with dependencies
Adding in variants from /tmp/tmp0cnnyrlc/info/recipe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /tmp/tmp0cnnyrlc/info/recipe/conda_build_config.yaml
Nothing to test for: /home/travis/miniconda/conda-bld/linux-64/pyocct-7.2.0.1-py36_0.tar.bz2

So it looks like it didn't find anything to test.

Started a new conda environment and installed the latest build. (I was careful to delete all traces of the earlier build, since conda tries very hard to reuse cached packages).

$ conda install -c trelau/label/master pyocct
<loading messages>
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
$ cd test
$ python -m unittest test_graphic3d.py
.......................EE.......
======================================================================
ERROR: test_THE_DEFAULT_DEPTH (test_graphic3d.TestGraphic3dRenderingParams)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_graphic3d.py", line 141, in test_THE_DEFAULT_DEPTH
    self.assertEqual(Graphic3d_RenderingParams.THE_DEFAULT_DEPTH, 3)
AttributeError: type object 'OCCT.Graphic3d.Graphic3d_RenderingParams' has no attribute 'THE_DEFAULT_DEPTH'

======================================================================
ERROR: test_THE_DEFAULT_RESOUTION (test_graphic3d.TestGraphic3dRenderingParams)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_graphic3d.py", line 138, in test_THE_DEFAULT_RESOUTION
    self.assertEqual(Graphic3d_RenderingParams.THE_DEFAULT_RESOLUTION, 72)
AttributeError: type object 'OCCT.Graphic3d.Graphic3d_RenderingParams' has no attribute 'THE_DEFAULT_RESOLUTION'

----------------------------------------------------------------------
Ran 32 tests in 0.004s

FAILED (errors=2)

So that's looking good. It appears that the Graphic3d_RenderingParams data fields are behaving properly and it's just the static attributes that are a problem. It would be interesting to work out why, but it's unlikely that they will be needed in practice, and can probably just be omitted.

I also testing importing all the modules:

$ python -m unittest test_modules.py 
.....................................................FF...............................FFFFF................................................................................................F.......F............................FFFF.......................F.............................................................x.....F.......................
<details omitted>
----------------------------------------------------------------------
Ran 343 tests in 5.871s

FAILED (failures=15, expected failures=1)

Again looks good. I assume the failures are expected at this stage, but I can post the details if it would be helpful.

[trelau]

@shrdluk hmm...i get only one expected failure on the test_modules.py script...could you post the name of the failed modules and any relevant errors? @looooo do you get same results?

[shrdluk]

$ python -m unittest test_modules.py 
.....................................................FF...............................FFFFF................................................................................................F.......F............................FFFF.......................F.............................................................x.....F.......................
======================================================================
FAIL: test_BaseTraceCollector (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 153, in test_BaseTraceCollector
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_BasicsGenericDestructor (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 161, in test_BasicsGenericDestructor
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_Driver (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 721, in test_Driver
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverDAT (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 729, in test_DriverDAT
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverGMF (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 737, in test_DriverGMF
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverSTL (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 745, in test_DriverSTL
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverUNV (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 753, in test_DriverUNV
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_LocalTraceBufferPool (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1521, in test_LocalTraceBufferPool
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_NETGENPlugin (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1617, in test_NETGENPlugin
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMDS (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1953, in test_SMDS
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMDSAbs (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1961, in test_SMDSAbs
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMESH (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1969, in test_SMESH
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMESHDS (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1977, in test_SMESHDS
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_StdMeshers (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 2025, in test_StdMeshers
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_WNT (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 2610, in test_WNT
    self.assertTrue(mod_ok)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 343 tests in 21.766s

FAILED (failures=15, expected failures=1)

[trelau]

@shrdluk oh all of the SMESH modules which are turned off. i think i'll move this script into the misc folder or something..it was mostly for debugging.

@looooo i see smesh is only for py36 on conda-forge, can the recipe be updated for py35?

[looooo]

I included the PR, but the problem is that (at least for me) currently the conda-smithy rerendering is broken. I guess re-rendering is necessary to add py35... So I think it's best to wait for the next conda-smithy release...

[shrdluk]

New conda environment.

$ conda install -c trelau/label/master pyocct
<installation messages, including:>
...
    netgen:           6.2.1804-py36_0         conda-forge        
    occt:             7.2.0-3                 conda-forge   
    pyocct:           7.2.0.1-py36_0          trelau/label/master
    smesh:            8.3.0b0-py36_5          conda-forge    
    vtk:              8.1.0-py36h969b564_201  
...

Ran test_modules.py. I know you've moved this out of the test directory, but it's still a good general health check to find any missing modules or undefined symbols.

$ python -m unittest test_modules.py
......................................................................................FFFFF........................................................................................................F............................F.FF.......................F.............................................................x.....F.......................
======================================================================
FAIL: test_Driver (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 721, in test_Driver
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverDAT (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 729, in test_DriverDAT
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverGMF (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 737, in test_DriverGMF
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverSTL (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 745, in test_DriverSTL
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_DriverUNV (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 753, in test_DriverUNV
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_NETGENPlugin (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1617, in test_NETGENPlugin
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMDS (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1953, in test_SMDS
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMESH (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1969, in test_SMESH
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_SMESHDS (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 1977, in test_SMESHDS
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_StdMeshers (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 2025, in test_StdMeshers
    self.assertTrue(mod_ok)
AssertionError: False is not true

======================================================================
FAIL: test_WNT (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/test/test_modules.py", line 2610, in test_WNT
    self.assertTrue(mod_ok)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 343 tests in 7.002s

FAILED (failures=11, expected failures=1)

Still a few errors, but these may not be relevant.

[trelau]

looks like all the SMESH modules still 🤔 i added import tests on the latest commit to see if they can be imported after the build...

[looooo]

the import work for me. Only OCCT.WNT seems to fail:

>>> import OCCT.WNT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'OCCT.WNT'

tested with occt from cad-channel and from trelau-channel.

[looooo]

@shrdluk I guess in your case smesh is not working because you are using vtk from defaults. Defaults should be compatible with conda-forge, but currently (at least for vtk) this isn't true. I remeber building smesh with vtk from defaults wasn't possible.
so maybe try: conda install vtk -c conda-forge

[shrdluk]

Yes, that seems to be it. The version of vtk in my conda environment was:

vtk                       8.1.0           py36h969b564_201

This is the version which conda pulls in when installing pyOCCT in a new environment.

After:

$ conda install vtk -c conda-forge

the version is:

vtk                       8.1.0           py36h99a48a5_205    conda-forge

And the result of running the tests:

$ python -m unittest test_modules.py 
.........................................................................................................................................................................................................................................................................................................................x.....F.......................
======================================================================
FAIL: test_WNT (test_modules.TestModuleImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/keith/projects/occt/pyOCCT/misc/test_modules.py", line 2610, in test_WNT
    self.assertTrue(mod_ok)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 343 tests in 9.853s

FAILED (failures=1, expected failures=1)

Is it possible to specify the channel in the meta.yaml build/run requirements to make sure the correct versions of other libraries are installed?

[looooo]

Is it possible to specify the channel in the meta.yaml build/run requirements to make sure the correct versions of other libraries are installed?

the simplest way is adding the channel to the .conda.rc file:

conda config --add channels conda-forge
conda config --add channels trelau

This will reult in a channel-section looking like this:

trelau
conda-forge
defaults

This means packages from trelau will be preferred over packages from conda-forge. And conda-forge packages will be preferred over defaults...

now creating a new env with:

conda create -n pyocct pyocct

will pull the right dependencies.

[shrdluk]

Ok, that's fine. This probably needs to be documented somewhere so that new users get their channels in the right order. I imagine it won't be a problem once pyOCCT builds become available from conda-forge.

[trelau]

I will add a note. without modifying conda.rc you could do:

conda install -c conda-forge -c trelau pyocct

i think. eventually conda install -c conda-forge pyocct