kat-plot_density.o: In function `boost::filesystem::exists(boost::filesystem::path const&)': /usr/include/boost/filesystem/operations.hpp:446: undefined reference to `boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*)'

[tgac]

kat-plot_density.o: In function `boost::filesystem::exists(boost::filesystem::path const&)': /usr/include/boost/filesystem/operations.hpp:446: undefined reference to `boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*)'

[maplesond]

Hmm, there is some issue detecting the filesystem and program_options static libraries. I configured KAT for static linking to this as we were getting user's encountering linking issues at runtime.

Here's what I get when I run grep BOOST src/Makefile in my environment:

BOOST_CHRONO_LIB = -lboost_chrono
BOOST_CHRONO_STATIC_LIB = /usr/lib/x86_64-linux-gnu/libboost_chrono.a
BOOST_CPPFLAGS = -I/usr/include
BOOST_FILESYSTEM_LIB = -lboost_filesystem
BOOST_FILESYSTEM_STATIC_LIB = /usr/lib/x86_64-linux-gnu/libboost_filesystem.a
BOOST_LDFLAGS = -L/usr/lib/x86_64-linux-gnu
BOOST_PROGRAM_OPTIONS_LIB = -lboost_program_options
BOOST_PROGRAM_OPTIONS_STATIC_LIB = /usr/lib/x86_64-linux-gnu/libboost_program_options.a
BOOST_STATIC_LIBS = /usr/lib/x86_64-linux-gnu/libboost_timer.a /usr/lib/x86_64-linux-gnu/libboost_chrono.a -lrt /usr/lib/x86_64-linux-gnu/libboost_filesystem.a /usr/lib/x86_64-linux-gnu/libboost_program_options.a /usr/lib/x86_64-linux-gnu/libboost_system.a
BOOST_SYSTEM_LIB = -lboost_system
BOOST_SYSTEM_STATIC_LIB = /usr/lib/x86_64-linux-gnu/libboost_system.a
BOOST_TIMER_LIB = -lboost_timer
BOOST_TIMER_STATIC_LIB = /usr/lib/x86_64-linux-gnu/libboost_timer.a

Can you verify that these static libraries exist?

[mmokrejs]

# equery files boost |  grep libboost
/usr/lib32/libboost_atomic-mt.so
/usr/lib32/libboost_atomic.so
/usr/lib32/libboost_atomic.so.1.62.0
/usr/lib32/libboost_chrono-mt.so
/usr/lib32/libboost_chrono.a
/usr/lib32/libboost_chrono.so
/usr/lib32/libboost_chrono.so.1.62.0
/usr/lib32/libboost_container-mt.so
/usr/lib32/libboost_container.so
/usr/lib32/libboost_container.so.1.62.0
/usr/lib32/libboost_context-mt.so
/usr/lib32/libboost_context.so
/usr/lib32/libboost_context.so.1.62.0
/usr/lib32/libboost_date_time-mt.so
/usr/lib32/libboost_date_time.so
/usr/lib32/libboost_date_time.so.1.62.0
/usr/lib32/libboost_exception.a
/usr/lib32/libboost_fiber-mt.so
/usr/lib32/libboost_fiber.so
/usr/lib32/libboost_fiber.so.1.62.0
/usr/lib32/libboost_filesystem-mt.so
/usr/lib32/libboost_filesystem.so
/usr/lib32/libboost_filesystem.so.1.62.0
/usr/lib32/libboost_graph-mt.so
/usr/lib32/libboost_graph.so
/usr/lib32/libboost_graph.so.1.62.0
/usr/lib32/libboost_graph_parallel-mt.so
/usr/lib32/libboost_graph_parallel.so
/usr/lib32/libboost_graph_parallel.so.1.62.0
/usr/lib32/libboost_iostreams-mt.so
/usr/lib32/libboost_iostreams.so
/usr/lib32/libboost_iostreams.so.1.62.0
/usr/lib32/libboost_locale-mt.so
/usr/lib32/libboost_locale.so
/usr/lib32/libboost_locale.so.1.62.0
/usr/lib32/libboost_log-mt.so
/usr/lib32/libboost_log.so
/usr/lib32/libboost_log.so.1.62.0
/usr/lib32/libboost_log_setup-mt.so
/usr/lib32/libboost_log_setup.so
/usr/lib32/libboost_log_setup.so.1.62.0
/usr/lib32/libboost_math_c99-mt.so
/usr/lib32/libboost_math_c99.so
/usr/lib32/libboost_math_c99.so.1.62.0
/usr/lib32/libboost_math_c99f-mt.so
/usr/lib32/libboost_math_c99f.so
/usr/lib32/libboost_math_c99f.so.1.62.0
/usr/lib32/libboost_math_c99l-mt.so
/usr/lib32/libboost_math_c99l.so
/usr/lib32/libboost_math_c99l.so.1.62.0
/usr/lib32/libboost_math_tr1-mt.so
/usr/lib32/libboost_math_tr1.so
/usr/lib32/libboost_math_tr1.so.1.62.0
/usr/lib32/libboost_math_tr1f-mt.so
/usr/lib32/libboost_math_tr1f.so
/usr/lib32/libboost_math_tr1f.so.1.62.0
/usr/lib32/libboost_math_tr1l-mt.so
/usr/lib32/libboost_math_tr1l.so
/usr/lib32/libboost_math_tr1l.so.1.62.0
/usr/lib32/libboost_mpi-mt.so
/usr/lib32/libboost_mpi.so
/usr/lib32/libboost_mpi.so.1.62.0
/usr/lib32/libboost_prg_exec_monitor-mt.so
/usr/lib32/libboost_prg_exec_monitor.so
/usr/lib32/libboost_prg_exec_monitor.so.1.62.0
/usr/lib32/libboost_program_options-mt.so
/usr/lib32/libboost_program_options.so
/usr/lib32/libboost_program_options.so.1.62.0
/usr/lib32/libboost_random-mt.so
/usr/lib32/libboost_random.so
/usr/lib32/libboost_random.so.1.62.0
/usr/lib32/libboost_regex-mt.so
/usr/lib32/libboost_regex.so
/usr/lib32/libboost_regex.so.1.62.0
/usr/lib32/libboost_serialization-mt.so
/usr/lib32/libboost_serialization.so
/usr/lib32/libboost_serialization.so.1.62.0
/usr/lib32/libboost_signals-mt.so
/usr/lib32/libboost_signals.so
/usr/lib32/libboost_signals.so.1.62.0
/usr/lib32/libboost_system-mt.so
/usr/lib32/libboost_system.a
/usr/lib32/libboost_system.so
/usr/lib32/libboost_system.so.1.62.0
/usr/lib32/libboost_test_exec_monitor.a
/usr/lib32/libboost_thread-mt.so
/usr/lib32/libboost_thread.so
/usr/lib32/libboost_thread.so.1.62.0
/usr/lib32/libboost_timer-mt.so
/usr/lib32/libboost_timer.a
/usr/lib32/libboost_timer.so
/usr/lib32/libboost_timer.so.1.62.0
/usr/lib32/libboost_type_erasure-mt.so
/usr/lib32/libboost_type_erasure.so
/usr/lib32/libboost_type_erasure.so.1.62.0
/usr/lib32/libboost_unit_test_framework-mt.so
/usr/lib32/libboost_unit_test_framework.a
/usr/lib32/libboost_unit_test_framework.so
/usr/lib32/libboost_unit_test_framework.so.1.62.0
/usr/lib32/libboost_wave-mt.so
/usr/lib32/libboost_wave.so
/usr/lib32/libboost_wave.so.1.62.0
/usr/lib32/libboost_wserialization-mt.so
/usr/lib32/libboost_wserialization.so
/usr/lib32/libboost_wserialization.so.1.62.0
/usr/lib64/libboost_atomic-mt.so
/usr/lib64/libboost_atomic.so
/usr/lib64/libboost_atomic.so.1.62.0
/usr/lib64/libboost_chrono-mt.so
/usr/lib64/libboost_chrono.a
/usr/lib64/libboost_chrono.so
/usr/lib64/libboost_chrono.so.1.62.0
/usr/lib64/libboost_container-mt.so
/usr/lib64/libboost_container.so
/usr/lib64/libboost_container.so.1.62.0
/usr/lib64/libboost_context-mt.so
/usr/lib64/libboost_context.so
/usr/lib64/libboost_context.so.1.62.0
/usr/lib64/libboost_date_time-mt.so
/usr/lib64/libboost_date_time.so
/usr/lib64/libboost_date_time.so.1.62.0
/usr/lib64/libboost_exception.a
/usr/lib64/libboost_fiber-mt.so
/usr/lib64/libboost_fiber.so
/usr/lib64/libboost_fiber.so.1.62.0
/usr/lib64/libboost_filesystem-mt.so
/usr/lib64/libboost_filesystem.so
/usr/lib64/libboost_filesystem.so.1.60.0
/usr/lib64/libboost_filesystem.so.1.62.0
/usr/lib64/libboost_graph-mt.so
/usr/lib64/libboost_graph.so
/usr/lib64/libboost_graph.so.1.62.0
/usr/lib64/libboost_graph_parallel-mt.so
/usr/lib64/libboost_graph_parallel.so
/usr/lib64/libboost_graph_parallel.so.1.62.0
/usr/lib64/libboost_iostreams-mt.so
/usr/lib64/libboost_iostreams.so
/usr/lib64/libboost_iostreams.so.1.60.0
/usr/lib64/libboost_iostreams.so.1.62.0
/usr/lib64/libboost_locale-mt.so
/usr/lib64/libboost_locale.so
/usr/lib64/libboost_locale.so.1.62.0
/usr/lib64/libboost_log-mt.so
/usr/lib64/libboost_log.so
/usr/lib64/libboost_log.so.1.62.0
/usr/lib64/libboost_log_setup-mt.so
/usr/lib64/libboost_log_setup.so
/usr/lib64/libboost_log_setup.so.1.62.0
/usr/lib64/libboost_math_c99-mt.so
/usr/lib64/libboost_math_c99.so
/usr/lib64/libboost_math_c99.so.1.62.0
/usr/lib64/libboost_math_c99f-mt.so
/usr/lib64/libboost_math_c99f.so
/usr/lib64/libboost_math_c99f.so.1.62.0
/usr/lib64/libboost_math_c99l-mt.so
/usr/lib64/libboost_math_c99l.so
/usr/lib64/libboost_math_c99l.so.1.62.0
/usr/lib64/libboost_math_tr1-mt.so
/usr/lib64/libboost_math_tr1.so
/usr/lib64/libboost_math_tr1.so.1.62.0
/usr/lib64/libboost_math_tr1f-mt.so
/usr/lib64/libboost_math_tr1f.so
/usr/lib64/libboost_math_tr1f.so.1.62.0
/usr/lib64/libboost_math_tr1l-mt.so
/usr/lib64/libboost_math_tr1l.so
/usr/lib64/libboost_math_tr1l.so.1.62.0
/usr/lib64/libboost_mpi-mt.so
/usr/lib64/libboost_mpi.so
/usr/lib64/libboost_mpi.so.1.62.0
/usr/lib64/libboost_mpi_python-2.7-mt.so
/usr/lib64/libboost_mpi_python-2.7.so
/usr/lib64/libboost_mpi_python-2.7.so.1.62.0
/usr/lib64/libboost_mpi_python-3.5-mt.so
/usr/lib64/libboost_mpi_python-3.5.so
/usr/lib64/libboost_mpi_python-3.5.so.1.62.0
/usr/lib64/libboost_prg_exec_monitor-mt.so
/usr/lib64/libboost_prg_exec_monitor.so
/usr/lib64/libboost_prg_exec_monitor.so.1.62.0
/usr/lib64/libboost_program_options-mt.so
/usr/lib64/libboost_program_options.so
/usr/lib64/libboost_program_options.so.1.60.0
/usr/lib64/libboost_program_options.so.1.62.0
/usr/lib64/libboost_python-2.7-mt.so
/usr/lib64/libboost_python-2.7.so
/usr/lib64/libboost_python-2.7.so.1.62.0
/usr/lib64/libboost_python-3.5-mt.so
/usr/lib64/libboost_python-3.5.so
/usr/lib64/libboost_python-3.5.so.1.62.0
/usr/lib64/libboost_random-mt.so
/usr/lib64/libboost_random.so
/usr/lib64/libboost_random.so.1.62.0
/usr/lib64/libboost_regex-mt.so
/usr/lib64/libboost_regex.so
/usr/lib64/libboost_regex.so.1.62.0
/usr/lib64/libboost_serialization-mt.so
/usr/lib64/libboost_serialization.so
/usr/lib64/libboost_serialization.so.1.62.0
/usr/lib64/libboost_signals-mt.so
/usr/lib64/libboost_signals.so
/usr/lib64/libboost_signals.so.1.62.0
/usr/lib64/libboost_system-mt.so
/usr/lib64/libboost_system.a
/usr/lib64/libboost_system.so
/usr/lib64/libboost_system.so.1.60.0
/usr/lib64/libboost_system.so.1.62.0
/usr/lib64/libboost_test_exec_monitor.a
/usr/lib64/libboost_thread-mt.so
/usr/lib64/libboost_thread.so
/usr/lib64/libboost_thread.so.1.62.0
/usr/lib64/libboost_timer-mt.so
/usr/lib64/libboost_timer.a
/usr/lib64/libboost_timer.so
/usr/lib64/libboost_timer.so.1.62.0
/usr/lib64/libboost_type_erasure-mt.so
/usr/lib64/libboost_type_erasure.so
/usr/lib64/libboost_type_erasure.so.1.62.0
/usr/lib64/libboost_unit_test_framework-mt.so
/usr/lib64/libboost_unit_test_framework.a
/usr/lib64/libboost_unit_test_framework.so
/usr/lib64/libboost_unit_test_framework.so.1.62.0
/usr/lib64/libboost_wave-mt.so
/usr/lib64/libboost_wave.so
/usr/lib64/libboost_wave.so.1.62.0
/usr/lib64/libboost_wserialization-mt.so
/usr/lib64/libboost_wserialization.so
/usr/lib64/libboost_wserialization.so.1.62.0

So, I do not have some of the static libs. I will probably need to ask other Gentoo Linux developers.

# equery files boost |  grep libboost | grep libboost_chrono.a
/usr/lib32/libboost_chrono.a
/usr/lib64/libboost_chrono.a
# equery files boost |  grep libboost | grep libboost_filesystem.a
# equery files boost |  grep libboost | grep libboost_program_options.a
# equery files boost |  grep libboost | grep libboost_system.a
/usr/lib32/libboost_system.a
/usr/lib64/libboost_system.a
# equery files boost |  grep libboost | grep libboost_timer.a
/usr/lib32/libboost_timer.a
/usr/lib64/libboost_timer.a
# 

It is unfortunate to require static libs. At least the shared libs should have been used for those missing static-ones. :(

https://forums.gentoo.org/viewtopic-t-730625-start-0.html

[maplesond]

Interesting that those libraries don't come bundled. I'll modify the configuration code so that it will prefer static libs if available but revert to shared if they are not.

In the meantime, instead of asking the gentoo develoeprs you can compile your own custom boost by doing something like this:

Download and extract the boost source code then do this (assuming the process hasn't changed):

./bootstrap.sh --with-libraries=chrono,timer,program_options,filesystem,system
./b2 -d0 install

[maplesond]

Also you can point configure to the custom boost directory with the --with-boost option.

[mmokrejs]

Thank you but I am trying to make a package for KAT to be included in Gentoo ( https://cgit.gentoo.org/proj/sci.git/commit/sci-biology/kat ), so I need to find a way to integrate it using standard packages.

Have just opened https://bugs.gentoo.org/show_bug.cgi?id=601914

[maplesond]

Ok cool. Thanks very much for doing this.

I'll get a fix for KAT for the next release. Out of interest what do you think the best strategy would be? Should I use all shared libraries if one or more static libs are missing, or should I proceed on a module by module basis?

[mmokrejs]

Either use just the shared libs (as you say, when at leats one static lib is missing) or exit during configure time. But requiring static libs is discouraged for a reason, even for security reasons and it would be hard to get the package into Gentoo.

[maplesond]

Ok, I'll do as you suggest here and revert to shared libraries for boost if one or more static libraries are missing.

Just out of interest what would be the security concerns with using static libraries? I thought the issues with static libraries were more down to forced loading of content which may not be required and an increase in binary size.

[maplesond]

I've made some changes to the develop branch that should do what we discussed. Could you double check that these work as expected for you before I create a new release?

[mmokrejs]

$ CFLAGS="${CFLAGS} -O2 -pipe -march=native" CXXFLAGS="${CFLAGS} -O2 -pipe -march=native" ./configure
...
checking for python... /scratch/mmokrejs/gentoo_rap/usr/bin/python
configure: Found python version: 2.7.12
checking for a version of Python >= '2.1.0'... yes
checking for a version of Python >= '3.1'... no
configure: WARNING: this package requires Python >= '3.1'.
If you have it installed, but it isn't the default Python
interpreter in your system path, please pass the PYTHON_VERSION
variable to configure. See ``configure --help'' for reference.

configure: Ensuring environment variable 'PYTHON_VERSION' set to: 
checking for the distutils Python package... yes
checking for Python include path... -I/scratch/mmokrejs/gentoo_rap/usr/include/python2.7
checking for Python library path... -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython2.7
checking for Python site-packages path... /scratch/mmokrejs/gentoo_rap/usr/lib64/python2.7/site-packages
checking python extra libraries... -lpthread -ldl  -lutil -lm
checking python extra linking flags... -Xlinker -export-dynamic
checking consistency of all components of python development environment... yes
checking for gnuplot... yes
checking for sphinx-build... yes
checking for boostlib >= 1.52... yes
checking whether the Boost::Filesystem library is available... yes
configure: WARNING: Could not find a dynamic version of the library!
configure: error: Could not find any version of the library to link to
$

First of all, I would propose configure to check the selected compiler works with the CXXFLAGS, actually with the merged kat_CXXFLAGS values. I was willing to test clang and icc compilers as well but I did not get to it because of the Boost shared lib assue above. So, will do later. ;-)

Here is the relevant snippet from config.log:

configure:18064: checking for python
configure:18082: found /scratch/mmokrejs/gentoo_rap/usr/bin/python
configure:18094: result: /scratch/mmokrejs/gentoo_rap/usr/bin/python
configure:18124: Found python version: 2.7.12
configure:18126: checking for a version of Python >= '2.1.0'
configure:18149: result: yes
configure:18160: checking for a version of Python >= '3.1'
configure:18174: result: no
configure:18180: WARNING: this package requires Python >= '3.1'.
If you have it installed, but it isn't the default Python
interpreter in your system path, please pass the PYTHON_VERSION
variable to configure. See ``configure --help'' for reference.

configure:18190: Ensuring environment variable 'PYTHON_VERSION' set to: 
configure:18196: checking for the distutils Python package
configure:18200: result: yes
configure:18219: checking for Python include path
configure:18235: result: -I/scratch/mmokrejs/gentoo_rap/usr/include/python2.7
configure:18242: checking for Python library path
configure:18319: result: -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython2.7
configure:18326: checking for Python site-packages path
configure:18332: result: /scratch/mmokrejs/gentoo_rap/usr/lib64/python2.7/site-packages
configure:18339: checking python extra libraries
configure:18346: result: -lpthread -ldl  -lutil -lm
configure:18353: checking python extra linking flags
configure:18360: result: -Xlinker -export-dynamic
configure:18367: checking consistency of all components of python development environment
configure:18395: gcc -o conftest  -O2 -pipe -march=native  -I/scratch/mmokrejs/gentoo_rap/usr/include/python2.7  -Xlinker -export-dynamic conftest.c  -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython2.7 -lpthread -ldl  -lutil -lm -lpthread -ldl  -lutil -lm >&5
configure:18395: $? = 0
configure:18413: result: yes
configure:18579: checking for gnuplot
configure:18595: found /scratch/mmokrejs/gentoo_rap/usr/bin/gnuplot
configure:18607: result: yes
configure:18633: checking for sphinx-build
configure:18649: found /scratch/mmokrejs/gentoo_rap/usr/bin/sphinx-build
configure:18661: result: yes
configure:18747: checking for boostlib >= 1.52
configure:18830: g++ -c  -O2 -pipe -march=native -O2 -pipe -march=native -std=gnu++11   conftest.cpp >&5
conftest.cpp:37:0: warning: "HAVE_PYTHON" redefined
 #define HAVE_PYTHON 0
 ^
conftest.cpp:36:0: note: this is the location of the previous definition
 #define HAVE_PYTHON "2.7"
 ^
configure:18830: $? = 0
configure:18832: result: yes
configure:19022: checking whether the Boost::Filesystem library is available
configure:19046: g++ -c  -O2 -pipe -march=native -O2 -pipe -march=native -std=gnu++11   conftest.cpp >&5
conftest.cpp:37:0: warning: "HAVE_PYTHON" redefined
 #define HAVE_PYTHON 0
 ^
conftest.cpp:36:0: note: this is the location of the previous definition
 #define HAVE_PYTHON "2.7"
 ^
configure:19046: $? = 0
configure:19060: result: yes
configure:19259: WARNING: Could not find a dynamic version of the library!
configure:19266: error: Could not find any version of the library to link to

So, I do not know which shared library it did not find. Do you see any missing from the listing in my comment few days ago? Sorry, do not see the comments numbered on github so I could refer to it easily.

[mmokrejs]

Just out of interest what would be the security concerns with using static libraries? I thought the
issues with static libraries were more down to forced loading of content which may not be required
and an increase in binary size.

Imagine boost had to be upgraded due to a security issue. You do the upgrade but forget that KAT has a static copy of the broken library ... It makes maintenance unnecessarily difficult. Also, static linking is tricky if pam modules, console-kit, -rpath etc. come into play.

Indeed, the loading is sometimes unnecessary and also, loader loads same library multiple times which is a waste, but I did not even mean this as the main concern.

[maplesond]

Re: static vs dynamic, that makes sense. I think in most KAT user environments (i.e HPC clusters, etc) the security issue shouldn't be too much of a concern. Normally these are not directly connected to the internet. But I can imagine that for internet exposed systems, and especially if made directly available via gentoo, then it is an issue. How about I also add a configure option so you can manually select to use dynamic boost libs?

With regards to the errors it seems there's 2 issues here.

1. Python issue. If you want to disable python plotting then I added an option to disable that completely: ./configure --disable-plotting. Alternatively, to use gnuplot instead, ./configure --enable-gnuplot. The preferred method is to use python plotting though, so this requires you to set the PYTHON_VERSION variable to a version 3 as discussed previously.

2. Missing boost lib. Not sure what's happening here. It looks like it can't find the boost_filesystem dynamic library, but this was previously available on your system. I haven't changed the code that detects boost modules. Can you verify these are available?

[mmokrejs]

It is is the priority number 1 issue actually. I do not know of HPC center not connected to internet anyway. Firewal is not enough.

You have something against Gentoo?

Adding --enable-static would be indeed a good idea, it is commonly exposed by other configure scripts. Enabled dynamic building should be the default, though.

I thought you adjusted the configure so that is properlyy fall back to python-2.7 if it wants to, disabling whatever has to be disabled as a consequence (probably doing --disable-plotting effectively if gnuplot was not found as I read it).

Can't you just select automatically detected python3 executable, where $PATH resolved it to?

$ equery files boost | grep boost_filesystem | xargs ls -la
lrwxrwxrwx 1 mmokrejs mmokrejs     22 Nov  9 21:12 /scratch/mmokrejs/gentoo_rap/usr/lib64/libboost_filesystem-mt.so -> libboost_filesystem.so
lrwxrwxrwx 1 mmokrejs mmokrejs     29 Nov  9 21:12 /scratch/mmokrejs/gentoo_rap/usr/lib64/libboost_filesystem.so -> libboost_filesystem.so.1.62.0
-rwxr-xr-x 1 mmokrejs mmokrejs 127704 Nov  9 21:10 /scratch/mmokrejs/gentoo_rap/usr/lib64/libboost_filesystem.so.1.62.0
$

[maplesond]

I've got nothing against gentoo, it's a great distro, sorry if I gave that impression. It's just that KAT typically needs 60GB+ of RAM to work and I don't imagine too many people using KAT in desktop environments etc. So the typical user will be working on a server or in a HPC environment. Maybe my experience of HPC environments is different but normally for security reasons access to the internet is restricted. Also the software management is normally different also as multiple version of the same software have to be maintained. This is also why it's more convenient for users to have reduced runtime dependencies in these environments.

With regards to python. KAT will NOT work with python2. The plotting scripts are very much python3 only. So there can be no fallback option here.

[mmokrejs]

Sorry, I forgot about the issue #61 in this regard but IMHO i is not "solved".

I can of course pass to configure PYTHON_VERSION=python3 but uhm, no, configure still provides the "python" substring so I may only do PYTHON_VERSION=3 ./configure . That is suboptimal. I cannot use a different python implementation and second, configure should itself figure out there is 'python3' executable in my $PATH.

[mmokrejs]

On another host I get:

checking for boostlib >= 1.52... yes
checking whether the Boost::Filesystem library is available... yes
checking for exit in -lboost_filesystem-mt... yes
configure: WARNING: Could not find a static version of the library!
checking whether the Boost::System library is available... yes
checking for exit in -lboost_system-mt... yes
checking for /usr/lib64/libboost_system.a... yes
checking whether the Boost::Program_Options library is available... yes
checking for exit in -lboost_program_options-mt... yes
checking whether the Boost::Chrono library is available... yes
checking for exit in -lboost_chrono-mt... yes
checking for /usr/lib64/libboost_chrono.a... yes
checking whether the Boost::Timer library is available... yes
checking for exit in -lboost_timer-mt... yes
checking for /usr/lib64/libboost_timer.a... yes
checking that generated files are newer than configure... done
configure: creating ./config.status

Sounds the WARNING: Could not find a static version of the library! is a false alarm, and it should include a library name it was looking for. Furher, the '!' does not make sense togetehr with WARNING. Warning are non-fatal, so I woul drop the '!'.

[mmokrejs]

The compile process on the other host failed while llinking, but I see now what is the problem. Some variables were not properly expanded:

/bin/sh ../libtool --tag=CXX --mode=link x86_64-pc-linux-gnu-g++ -g -O3 -fwrapv -Wall -Wextra -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-parameter -Wno-unused-variable -Wno-unused-command-line-argument -ansi -pedantic -std=c++11 -DCPLUSPLUS -O2 -pipe -maes -mpclmul -mpopcnt -mavx -march=native -std=gnu++11 -L../lib/.libs -L../deps/jellyfish-2.2.0/.libs -L/usr/lib64 -Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -o kat kat-plot_density.o kat-plot_profile.o kat-plot_spectra_cn.o kat-plot_spectra_mx.o kat-plot_spectra_hist.o kat-plot.o kat-filter_kmer.o kat-filter_sequence.o kat-filter.o kat-comp.o kat-gcp.o kat-histogram.o kat-sect.o kat-kat.o -pthread /usr/lib64/libboost_timer.a /usr/lib64/libboost_chrono.a -lrt /usr/lib64/libboost_system.a -l:libkat.a -l:libkat_jellyfish.a

[maplesond]

Ok, so I've added a configure option called --enable-dynamic-boost. I've done it this way because I don't want to blanket statically link, or dynamically link everything. Hopefully this compromise will suit both our use cases.

I've removed the exclamation marks from the end of the warning message and added the library. However, the warning messages are still present for now.

With python I've forced it to look for python3, so hopefully you shouldn't get any issues with python2. However, I'm basically reusing someone elses m4 macro for detecting python. Allowing a "python" prefix to the PYTHON_VERSION variable will mean changing quite a bit of code, which I'd like to avoid. It is also clearly specified in the configure help message that a version number is required rather than an executable name, and that this version number will be appended to the canonical python interpreter name. Unfortunately, this won't work with any other flavours of python that use a different executable name that isn't prefixed with "python".

If also think you just tested the master branch rather than the develop branch. Could check out the develop branch, and see if these new changes work? Thanks.

[mmokrejs]

Cool, a few tests using the develop banch now (sorry, I indeed used HEAD previously):

$ CFLAGS="${CFLAGS} -O2 -pipe -march=native" CXXFLAGS="${CFLAGS} -O2 -pipe -march=native" ./configure
...
checking for python3... /scratch/mmokrejs/gentoo_rap/usr/bin/python3
configure: Found python version: 3.4.4
checking for a version of Python >= '3.1.0'... yes
checking for a version of Python >= '3.1'... yes
configure: Ensuring environment variable 'PYTHON_VERSION' set to: 3
checking for the distutils Python package... yes
checking for Python include path... -I/scratch/mmokrejs/gentoo_rap/usr/include/python3.4m
checking for Python library path... -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython3.4m
checking for Python site-packages path... /scratch/mmokrejs/gentoo_rap/usr/lib64/python3.4/site-packages
checking python extra libraries... -lpthread -ldl  -lutil -lm
checking python extra linking flags... -Xlinker -export-dynamic
checking consistency of all components of python development environment... yes
checking python3 module: numpy... no
checking python3 module: matplotlib... no
checking python3 module: scipy... no
checking for gnuplot... yes
checking for sphinx-build... yes
checking for boostlib >= 1.52... yes
checking whether the Boost::Filesystem library is available... yes
configure: WARNING: Could not find a dynamic version of boost_filesystem
configure: error: Could not find any version of the library to link to

config.log contains:

configure:17991: checking for library containing clock_gettime
configure:18022: g++ -o conftest  -O2 -pipe -march=native -O2 -pipe -march=native -std=gnu++11   conftest.cpp  >&5
configure:18022: $? = 0
configure:18039: result: none required
configure:18084: checking for python3
configure:18102: found /scratch/mmokrejs/gentoo_rap/usr/bin/python3
configure:18114: result: /scratch/mmokrejs/gentoo_rap/usr/bin/python3
configure:18144: Found python version: 3.4.4
configure:18146: checking for a version of Python >= '3.1.0'
configure:18169: result: yes
configure:18180: checking for a version of Python >= '3.1'
configure:18187: result: yes
configure:18210: Ensuring environment variable 'PYTHON_VERSION' set to: 3
configure:18216: checking for the distutils Python package
configure:18220: result: yes
configure:18239: checking for Python include path
configure:18255: result: -I/scratch/mmokrejs/gentoo_rap/usr/include/python3.4m
configure:18262: checking for Python library path
configure:18339: result: -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython3.4m
configure:18346: checking for Python site-packages path
configure:18352: result: /scratch/mmokrejs/gentoo_rap/usr/lib64/python3.4/site-packages
configure:18359: checking python extra libraries
configure:18366: result: -lpthread -ldl  -lutil -lm
configure:18373: checking python extra linking flags
configure:18380: result: -Xlinker -export-dynamic
configure:18387: checking consistency of all components of python development environment
configure:18415: gcc -o conftest  -O2 -pipe -march=native  -I/scratch/mmokrejs/gentoo_rap/usr/include/python3.4m  -Xlinker -export-dynamic conftest.c  -L/scratch/mmokrejs/gentoo_rap/usr/lib64 -lpython3.4m -lpthread -ldl  -lutil -lm -lpthread -ldl  -lutil -lm >&5
configure:18415: $? = 0
configure:18433: result: yes
configure:18486: checking python3 module: numpy
configure:18497: result: no
configure:18524: checking python3 module: matplotlib
configure:18535: result: no
configure:18562: checking python3 module: scipy
configure:18573: result: no
configure:18599: checking for gnuplot
configure:18615: found /scratch/mmokrejs/gentoo_rap/usr/bin/gnuplot
configure:18627: result: yes
configure:18654: checking for sphinx-build
configure:18670: found /scratch/mmokrejs/gentoo_rap/usr/bin/sphinx-build
configure:18682: result: yes
configure:18778: checking for boostlib >= 1.52
configure:18861: g++ -c  -O2 -pipe -march=native -O2 -pipe -march=native -std=gnu++11   conftest.cpp >&5
conftest.cpp:37:0: warning: "HAVE_PYTHON" redefined
 #define HAVE_PYTHON 0
 ^
conftest.cpp:36:0: note: this is the location of the previous definition
 #define HAVE_PYTHON "3.4"
 ^
configure:18861: $? = 0
configure:18863: result: yes
configure:19053: checking whether the Boost::Filesystem library is available
configure:19077: g++ -c  -O2 -pipe -march=native -O2 -pipe -march=native -std=gnu++11   conftest.cpp >&5
conftest.cpp:37:0: warning: "HAVE_PYTHON" redefined
 #define HAVE_PYTHON 0
 ^
conftest.cpp:36:0: note: this is the location of the previous definition
 #define HAVE_PYTHON "3.4"
 ^
configure:19077: $? = 0
configure:19091: result: yes
configure:19290: WARNING: Could not find a dynamic version of boost_filesystem
configure:19297: error: Could not find any version of the library to link to

A small nit pick is that configure check for gettime contains my CXXFLAGS twice. Or is that merged CFLAGS with CXXFLAGS instead? That would be wrong, only CXXFLAGS should be sent to C++ compiler.

Thank you for adding the --enable-dynamic-boost but add also --enable-shared which is the one we will use in Gentoo as teh default. Also --enable-static should be available so users can override the defaults as usual on Gentoo. All classes we use/inherit anticipate these commonly used arguments and this should be the first choice for us.
I intend to test all combinations of these arguments of course, do not worry. Thank you for your great efforts.

[maplesond]

Ok, so it looks a bit cleaner now. I noticed that you do not have numpy, scipy and matplotlib on your system. I think the behaviour here will be to disable python plotting anyway. It has detected gnuplot however, so this will be used in its place.

You still have the issue with boost_filesystem not being detected but presumably this is an issue with your environment rather than KAT?

With regards to your CXXFLAGS, are you not explicitly requesting these flags to be added twice with your command: CFLAGS="${CFLAGS} -O2 -pipe -march=native" CXXFLAGS="${CFLAGS} -O2 -pipe -march=native" ./configure ?

In terms of --enable-shared and --enable-static these should already be available in configure. My understanding was that these options relate to creation of shared and static libraries rather than linking to them however. Anyway, type ./configure --help to see the full set of options. Most of these are created by autoconf but a few are added either by my m4 macros or directly in the configure.ac file.

[maplesond]

Actually, if you type ./configure --help=recursive then you get the jellyfish options as well. You might want to consider passing in --with-sse if you want to create an optimised version.

[mmokrejs]

I forgot to include these dependencies to the package definition file, fixed now. Also added the --with-sse so it is passed down to jellyfish's configure (untested yet). Yes, I had a copy-pasting-related error in the commandline so the $CFLAGS were really added twice because of this.

My understanding was that these options relate to creation of shared and static libraries rather than linking to them however.
I will have to read some docs, maybe you are correct and users are maybe supposed to add -Bstatic to their compiler flags to achieve the static linking.

The issue with missing boost_filesystem is weird. Could it be the check fails because some of the other boost libs are available only in shared variants? So that mixed linking is declined somewhere? I doubt it but do not have any other idea at the moment.

[maplesond]

It's a bit of a pain and I wouldn't normally recommend it but the boost macros I'm using should allow you to specify explicit boost libraries for each module.

e.g. --configure --with-boost-libdir=/usr/lib32 --with-boost-filesystem=libboost_filesystem-mt

I haven't tested this though. Just thought I'd draw your attention to it in case it helps.

[alexiswl]

Hello, I was able to install this successfully using the --with-boost-libdir and --with-boost flags.

I have these lines in my bash_profile so I am curious as to why I needed to add the two boost flags.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/boost_1_62_0/build/lib/
export C_INCLUDE_PATH=$C_INCLUDE_PATH:$HOME/boost_1_62_0/build/include/
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:$HOME/boost_1_62_0/build/include/

I also had to add $HOME/anaconda3/lib/ to my LD_LIBRARY_PATH.
After receiving an error in the 'make check' command.

I am surprised that a
'which python3'
~/anaconda3/bin/python3
does not infer the python library path.

Everything else ran smoothly.

Alexis.