Dockerfile for SRBench

[yoshitomo-matsubara]

Dockerfile for SRBench

[foolnotion]

There are two issues:

1. eigen is pinned to version 3.3.9 in environment.yml, but Ceres wants 3.4.0
2. your system's C++ compiler is too old (g++ 8.3.0)

These changes should fix the issues:

# git diff
diff --git a/environment.yml b/environment.yml
index 314bff5..f349fc0 100755
--- a/environment.yml
+++ b/environment.yml
@@ -2,9 +2,11 @@ name: srbench
 channels:
     - conda-forge
 dependencies:
+    - gcc
+    - gxx
     - python
     - cmake=3.19.1 # operon, feat 
-    - eigen=3.3.9 # operon, feat, ellyn
+    - eigen=3.4.0 # operon, feat, ellyn
     - boost=1.74.0 # gpgomea, ellyn
     - pkg-config # gpgomea
     - scikit-learn=0.24.1 
@@ -23,6 +25,7 @@ dependencies:
     - fmt=7.1.3 
     - ceres-solver=2.0.0 
     - tbb-devel=2020.2 
+    - taskflow=3.2.0
     - openlibm 
     #deep-symbolic-regression 
     - tensorflow=1.14 
diff --git a/experiment/methods/src/operon_install.sh b/experiment/methods/src/operon_install.sh
index d125abf..d64ce60 100755
--- a/experiment/methods/src/operon_install.sh
+++ b/experiment/methods/src/operon_install.sh
@@ -15,7 +15,7 @@ fi
 git clone https://github.com/heal-research/operon
 cd operon
 # fix version
-git checkout 015d420944a64353a37e0493ae9be74c645b4198
+# git checkout 015d420944a64353a37e0493ae9be74c645b4198
 
 # run cmake
 rm -rf build

[yoshitomo-matsubara]

@foolnotion Thank you for the update. As seen in the first Dockerfile, I tried to update gcc and libeigen myself, which didn't resolve the issue. It is interesting to see those were updated through conda while apt install with specified didn't update their versions.

With the above changes, operon was successfully installed, but it caused new errors for DSR and returned "2 installs failed: dsr_install.sh"
This is the error from DSR. dsr-error.txt
It looks like you didn't change anything in Dockerfile. Did you change anything else?
I built docker image from scratch as described above and then made the changes you suggested

[foolnotion]

I didn't change anything else. This looks like a linker error, which suggests that something is broken within the conda environment. My first suggestion would be nuke everything and do a fresh install. I found numerous instances of the same linker error on the internet but no clear solution.

EDIT: dsr installed successfully after updating conda and then updating the environment.

[yoshitomo-matsubara]

@foolnotion Thank you for the suggestion.
I think I resolved it by installing gcc, gxx, eigen, and taskflow to conda as part of operon_install.sh (see below). Closing this issue now.

# # Installing operon
# export CC=gcc-9
# export CXX=gcc-9
# # Dependencies
# # - TBB - installed via conda
# # - Eigen- installed via conda
# # - Ceres- installed via conda
# # - {fmt}- installed via conda

# # remove directory if it exists
if [ -d operon ]; then
    rm -rf operon
fi

# Added by Yoshitomo Matsubara
conda install -c conda-forge gcc gxx eigen=3.4.0 taskflow=3.2.0 -y

git clone https://github.com/heal-research/operon
cd operon
# fix version
# commented out as suggested https://github.com/EpistasisLab/srbench/issues/55#issuecomment-917067097
git checkout 015d420944a64353a37e0493ae9be74c645b4198

# run cmake
rm -rf build
mkdir build; cd build;
SOURCE_DATE_EPOCH=`date +%s` cmake .. -DCMAKE_BUILD_TYPE=Release  -DBUILD_PYBIND=ON -DUSE_OPENLIBM=ON -DUSE_SINGLE_PRECISION=ON -DCERES_TINY_SOLVER=ON -DPython3_FIND_VIRTUALENV=ONLY #-DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX/lib/python3.7/site-packages/ #-DPython3_EXECUTABLE=$CONDA_PREFIX/bin/python -DPython3_LIBRARY=$CONDA_PREFIX/lib/ -DPython3_INCLUDE_DIR=$CONDA_PREFIX/include/ 
#-DPython3_ROOT_DIR=$CONDA_PREFIX
# -DPython3_FIND_STRATEGY=LOCATION 
# build
make VERBOSE=1 -j pyoperon

# install
make install

[lacava]

I'm confused why Ceres is all the sudden complaining about the eigen version when eigen, ceres, and operon are all fixed versions. @foolnotion do you understand it? current master branch builds are starting to fail with this same error.

i'm ok to update operon but want to keep the version used in the latest experiment active in an upcoming release.

[foolnotion]

@lacava I believe the reason is because Ceres enforces (via its cmake scripts) that the same Eigen version is used (when building code including Ceres headers - Operon) as the version the Ceres library itself was built with. Since the conda-provided Ceres was built against Eigen 3.4, but the environment.yml pins Eigen at 3.3.9, the cmake script throws an error.

The fix is simple: bump Eigen to 3.4 in environment.yml, everything else can stay the same. I doubt this version bump can affect anything at all.

[lacava]

eigen 3.4 makes FEAT fail actually. some syntax change in Eigen. plus, we should be able to fix the environment to what was used to produce the results.

it turns out ceres doesn't update its version with releases. trying now to specify the version from 4 months ago https://anaconda.org/conda-forge/ceres-solver/files

[foolnotion]

It appears that conda can't handle transitive dependencies very well (or at all :) ). If using an older build of Ceres fixes this, then perhaps pin everything down with conda-lock to avoid future issues. The resulting rev of srbench will then be the reference revision for the NIPS paper.

[foolnotion]

@lacava FEAT is easily fixed:

diff --git a/src/pop/op/learn/n_fuzzy_fixed_split.cc b/src/pop/op/learn/n_fuzzy_fixed_split.cc
index f96716f4..6bd52412 100755
--- a/src/pop/op/learn/n_fuzzy_fixed_split.cc
+++ b/src/pop/op/learn/n_fuzzy_fixed_split.cc
@@ -245,8 +245,8 @@ namespace FT{
             {
                 VectorXf class_weights(uc.size());
                 for (auto c : uc){
-                    class_weights(c) = 0;
-                    class_weights(c) = float(
+                    class_weights((int)c) = 0;
+                    class_weights((int)c) = float(
                             (classes.cast<int>().array() == int(c)).count()
                             )/classes.size();
                     /* cout << "class_weights for " << c << ": " 
diff --git a/src/pop/op/learn/n_fuzzy_split.cc b/src/pop/op/learn/n_fuzzy_split.cc
index 12a78cd2..a2099fa9 100755
--- a/src/pop/op/learn/n_fuzzy_split.cc
+++ b/src/pop/op/learn/n_fuzzy_split.cc
@@ -229,8 +229,8 @@ namespace FT{
             {
                 VectorXf class_weights(uc.size());
                 for (auto c : uc){
-                    class_weights(c) = 0;
-                    class_weights(c) = float(
+                    class_weights((int)c) = 0;
+                    class_weights((int)c) = float(
                             (classes.cast<int>().array() == int(c)).count()
                             )/classes.size();
                     /* cout << "class_weights for " << c << ": " 
diff --git a/src/pop/op/learn/n_split.cc b/src/pop/op/learn/n_split.cc
index 8fc8b43d..551bbee8 100755
--- a/src/pop/op/learn/n_split.cc
+++ b/src/pop/op/learn/n_split.cc
@@ -208,8 +208,8 @@ namespace Op{
     {
         VectorXf class_weights(uc.size());
         for (auto c : uc){
-            class_weights(c) = 0;
-            class_weights(c) = float(
+            class_weights((int)c) = 0;
+            class_weights((int)c) = float(
                     (classes.cast<int>().array() == int(c)).count()
                     )/classes.size();
             /* cout << "class_weights for " << c << ": " 

You were using a float to index into a vector, I assume this was previously silently cast to an int, but newer Eigen leverages SFINAE to turn this into an error (as it should).

[lacava]

thanks @foolnotion, I'll update FEAT. For SRBench I'm pushing an update to the environment that specifies a specific ceres package from conda that was used a few months ago.

[lacava]

Hi @yoshitomo-matsubara, if you are interested in contributing a Dockerfile to this repo that would be great. I'll close this issue for now since that wasn't its original goal, but open a new one to that end. I think that might be a better approach than trying to fix/lock conda env for different OSes, although both could work...

[yoshitomo-matsubara]

Hi @lacava,

Sure, I'll send a PR with Dockerfile soon