nlesc-dirac/sagecal

SAGECal is a fast, memory efficient and GPU accelerated radio interferometric calibration program. It supports all source models including points, Gaussians and Shapelets. Distributed calibration using MPI and consensus optimization is enabled. Both spectral and spatial priors can be used as constraints. Tools to build/restore sky models are included.

sagecal's Issues

cmake build fails on Cent OS 7.4 on fs5 on DAS5

[hspreeuw@fs5 build]$ cmake .. -DCASACORE_INCLUDE_DIR=/cm/shared/package/casacore/v2.3.0-gcc-4.9.3/include/ -DWCSLIB_INCLUDE_DIR=/cm/shared/package/wcslib/5.16-gcc-4.9.3/include -DWCSLIB_LIBRARY=/cm/shared/package/wcslib/5.16-gcc-4.9.3/lib/libwcs.so -DCFITSIO_LIBRARY=/cm/shared/package/cfitsio/3410-gcc-4.9.3/lib/libcfitsio.so -DCFITSIO_INCLUDE_DIR=/cm/shared/package/cfitsio/3380-gcc-4.9.3/include/ -DHDF5_LIBRARY_DIRS=/cm/shared/package/hdf5/1.8.17/lib
......
make -j4
......
/home/hspreeuw/sagecal-code/src/MS/main.cpp:471:135: warning: ISO C++98 does not support the ‘%lf’ gnu_printf format [-Wformat=]
/home/hspreeuw/sagecal-code/src/MS/main.cpp:471:135: warning: ISO C++98 does not support the ‘%lf’ gnu_printf format [-Wformat=]
[100%] Linking CXX executable ../../dist/bin/sagecal
/cm/shared/package/gcc/4.9.3/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../../../x86_64-unknown-linux-gnu/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
make[2]: *** [dist/bin/sagecal] Error 1
make[1]: *** [src/MS/CMakeFiles/sagecal.dir/all] Error 2
make: *** [all] Error 2

Check cloudera Spark platform

file names are confusing

sometimes files are called like clmfit_nocuda.c
and some times like predict_withbeam_gpu.c

that should be consistent

code documentation is needed

create recipies to install sagecal on major linux distributions

cmake build fails on Fedora 27

in build directory:
rm -rf *
cmake .. -DENABLE_CUDA=OFF -DCASACORE_ROOT_DIR=........./Casacore/casacore-2.0.3
-- The CXX compiler identification is GNU 7.3.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is GNU 7.3.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done

############################

Building SageCal v0.1.0

############################
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.3.12")

############################

Searching Sagecal dependencies

############################
-- Found WCSLIB: /usr/lib64/libwcs.so
-- Found CFITSIO: /usr/lib64/libcfitsio.so (found version "3.370")
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- A library with BLAS API found.
-- Looking for cheev_
-- Looking for cheev_ - found
-- A library with LAPACK API found.
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib64/libhdf5.so;/usr/lib64/libsz.so;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.8.18")
-- Found the following Casacore components:
-- coordinates
-- images
-- msfits
-- lattices
-- mirlib
-- fits
-- ms
-- measures
-- scimath
-- tables
-- scimath_f
-- casa
-- A library with BLAS API found.
-- A library with LAPACK API found.
-- Found OpenBLAS libraries: /usr/lib64/libopenblas.so
-- Found OpenBLAS include: /usr/include
-- Checking for module 'glib-2.0'
-- Found glib-2.0, version 2.54.3

############################

Package summary

############################
-- CMAKE_SYSTEM .......... = Linux-4.15.3-300.fc27.x86_64
-- CMAKE_BUILD_TYPE ...... =
-- BUILD_SHARED_LIBS ..... =
-- CMAKE_INSTALL_NAME_DIR =
-- USE_STACKTRACE ........ =
-- CMAKE_CXX_COMPILER .... = /usr/bin/c++
-- CMAKE_CXX_FLAGS ....... =
-- ENABLE_MPI ............ = OFF
-- ENABLE_CUDA ........... = OFF
-- BLAS-LIBS ............. = /usr/lib64/libopenblas.so
-- GLIB-INC .............. = /usr/include/glib-2.0;/usr/lib64/glib-2.0/include
-- GLIB-LIBS ............. = glib-2.0
-- LAPACK-INC ............ =
-- LAPACK-LIBS ........... = /usr/lib64/libopenblas.so;/usr/lib64/libopenblas.so
-- CFITSIO-INC .......... = /usr/include/cfitsio
-- CFITSIO-LIBS .......... = /usr/lib64/libcfitsio.so;/usr/lib64/libm.so
-- WCSLIB-INC ........... = /usr/include
-- WCSLIB-LIBS ........... = /usr/lib64/libwcs.so;/usr/lib64/libm.so

Compiling lib/Dirac
-- Compiling lib/Dirac without CUDA support.
-- lib/Dirac objects --> admm_solve;clmfit_nocuda;manifold_average;mdl;myblas;robust_lbfgs_nocuda;robustlm;rtr_solve_robust_admm;updatenu;lmfit_nocuda;consensus_poly;lbfgs_nocuda;rtr_solve;rtr_solve_robust

Compiling lib/Radio
-- Compiling lib/Radio without CUDA support.
-- lib/Radio objects --> readsky;predict;residual;transforms;stationbeam;predict_withbeam
-- Configuring done
-- Generating done
-- Build files have been written to: ......./sagecal/build

make
Scanning dependencies of target robust_lbfgs_nocuda
[ 1%] Building C object src/lib/Dirac/CMakeFiles/robust_lbfgs_nocuda.dir/robust_lbfgs_nocuda.c.o
[ 3%] Linking C shared library /dist/lib/librobust_lbfgs_nocuda.so
/usr/bin/ld: cannot open output file /dist/lib/librobust_lbfgs_nocuda.so: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [src/lib/Dirac/CMakeFiles/robust_lbfgs_nocuda.dir/build.make:95: /dist/lib/librobust_lbfgs_nocuda.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:138: src/lib/Dirac/CMakeFiles/robust_lbfgs_nocuda.dir/all] Error 2

coding standard is needed

coding standard or at least editorconfig should be added

clang tools scan-build and scan-view detect multiple issues in building libdirac.a

Disregarding the dead assignments, below I will provide details on the other issues:

Bug Group | Bug Type ▾ | File | Line | Path Length

Unix API | Allocator sizeof operand mismatch | lbfgs_nocuda.c | 321 | 1 |
Unix API | Allocator sizeof operand mismatch | robust_lbfgs_nocuda.c | 168 | 1 |
Logic error | Assigned value is garbage or undefined | lmfit_nocuda.c | 1093 | 6 |
Dead store | Dead initialization | rtr_solve.c | 382 | 1
Dead store | Dead initialization | rtr_solve_robust.c | 564 | 1
Logic error | Dereference of null pointer | clmfit_nocuda.c | 890 | 31
Logic error | Result of operation is garbage or undefined | lmfit_nocuda.c | 443 | 7
Unix API | Undefined allocation of 0 bytes (CERT MEM04-C; CWE-131) | lbfgs_nocuda.c | 315 | 16
Unix API | Undefined allocation of 0 bytes (CERT MEM04-C; CWE-131) | robust_lbfgs_nocuda.c | 162 | 16

I can provide even more detail or you can extract that yourself by installing the clang static analysis package.

clang tools scan-build and scan-view detect multiple issues in building libradio.a

Only a single issue, except for the dead assignments, which are supposed to be false positives:

Logic error | Result of operation is garbage or undefined | readsky.c | 702 | 12

Build problem when not on the DAS5 fs5 headnode: make[2]: *** No rule to make target `/usr/lib64/libopenblas.so', needed by `dist/bin/sagecal'

Describe the bug
Cannot build SAGECal when libopenblas.so is not in /usr/lib64/

make[2]: *** No rule to make target /usr/lib64/libopenblas.so', needed by dist/bin/sagecal'. Stop.

To Reproduce
Steps to reproduce the behavior:

1. ssh to fs5 headnode on DAS5
2. ssh to node
3. cd sagecal-code/build
4. module load casacore/2.3.0-gcc-4.9.3
5. module load cmake/3.8.2
6. cmake .. -DCMAKE_INSTALL_PREFIX=/home/hspreeuw/sagecal-code/src/MS -DHAVE_CUDA=FALSE
7. make ---> No rule to make target `/usr/lib64/libopenblas.so'
8. ls /usr/lib64/libopenblas.so----> ls: cannot access /usr/lib64/libopenblas.so: No such file or directory. On the headnode, we do have a file /usr/lib64/libopenblas.so
9. module load openblas
10. cmake .. -DCMAKE_INSTALL_PREFIX=/home/hspreeuw/sagecal-code/src/MS -DHAVE_CUDA=FALSE
11. make---> No rule to make target `/usr/lib64/libopenblas.so'

Expected behavior
Successful build

Additional context
This is the gpu_support branch. Build for CPU.

Create a dev branch for the development version

Find a new filesystem

As HDFS is dead, we need a new filesystem to replace it.

install statements for all targets and SO versions for libs missing

Compilation issue on Ubuntu 18.04 with cuda enabled

[  1%] Building C object src/lib/Dirac/CMakeFiles/rtr_solve_robust_cuda.dir/rtr_solve_robust_cuda.c.o
cd /packaging/kern/packaging/build/sagecal/obj-x86_64-linux-gnu/src/lib/Dirac && /usr/bin/cc -Drtr_solve_robust_cuda_EXPORTS -I/usr/include/x86_64-linux-
gnu -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/packaging/kern/packaging/build/sagecal/src/lib/Dirac/.  -march=native -O2 -pi
pe -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -o CMakeFiles/rtr_solve_robust_cuda.dir/rtr_solve_robust_cuda.c.o   -c /packaging/kern/packaging/build/sagecal
/src/lib/Dirac/rtr_solve_robust_cuda.c
/packaging/kern/packaging/build/sagecal/src/lib/Dirac/rtr_solve_robust_cuda.c:45:18: error: unknown type name ‘cublasStatus_t’; did you mean ‘cudaStream_
t’?
 checkCublasError(cublasStatus_t cbstatus, char *file, int line)
                  ^~~~~~~~~~~~~~
                  cudaStream_t
/packaging/kern/packaging/build/sagecal/src/lib/Dirac/rtr_solve_robust_cuda.c:65:83: error: unknown type name ‘cuFloatComplex’; did you mean ‘complex’?
 cudakernel_fns_fgrad_robust(int ThreadsPerBlock, int BlocksPerGrid, int N, int M, cuFloatComplex *x, cuFloatComplex *eta, float *y, float *coh, short *b
bh, float *iw, float *wtd, int negate, cublasHandle_t cbhandle, cusolverDnHandle_t solver_handle) {
                                                                                   ^~~~~~~~~~~~~~
                                                                                   complex
/packaging/kern/packaging/build/sagecal/src/lib/Dirac/rtr_solve_robust_cuda.c:65:102: error: unknown type name ‘cuFloatComplex’; did you mean ‘complex’?
 cudakernel_fns_fgrad_robust(int ThreadsPerBlock, int BlocksPerGrid, int N, int M, cuFloatComplex *x, cuFloatComplex *eta, float *y, float *coh, short *b
bh, float *iw, float *wtd, int negate, cublasHandle_t cbhandle, cusolverDnHandle_t solver_handle) {
                                                                                                      ^~~~~~~~~~~~~~
                                                                                                      complex
/packaging/kern/packaging/build/sagecal/src/lib/Dirac/rtr_solve_robust_cuda.c:65:192: error: unknown type name ‘cublasHandle_t’
 nt BlocksPerGrid, int N, int M, cuFloatComplex *x, cuFloatComplex *eta, float *y, float *coh, short *bbh, float *iw, float *wtd, int negate, cublasHandl
e_t cbhandle, cusolverDnHandle_t solver_handle) {
                                                                                                                                              ^~~~~~~~~~~
~~~
/packaging/kern/packaging/build/sagecal/src/lib/Dirac/rtr_solve_robust_cuda.c:65:217: error: unknown type name ‘cusolverDnHandle_t’; did you mean ‘cudaIp
cMemHandle_t’?
 int M, cuFloatComplex *x, cuFloatComplex *eta, float *y, float *coh, short *bbh, float *iw, float *wtd, int negate, cublasHandle_t cbhandle, cusolverDnH
andle_t solver_handle) {
                                                                                                                                              ^~~~~~~~~~~
~~~~~~~

                                                                cudaIpcMemHandle_t

etc etc

add a debug build option which shows all the warnings

cmake build fails on Ubuntu 16.04

cmake .. -DENABLE_CUDA=OFF
make
---->

add clang to continuous integration

Does not build with CMake on MacOS

Two things are preventing sagecal to build on my machine (with cmake): myblas is not properly linked in, and openblas is not linked. CMake version is 3.8.2.

➜  build git:(openblas_root) ✗ cmake .. -DCASACORE_ROOT_DIR=~/opt/casacore -DOpenBLAS_ROOT_DIR=/usr/local/opt/openblas -DENABLE_CUDA=Off
-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is AppleClang 9.0.0.9000039
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
--
############################
#   Building SageCal v0.6.0
############################
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
--
############################
#   Searching Sagecal dependencies
############################
-- Found WCSLIB: /usr/local/lib/libwcs.dylib
-- Found CFITSIO: /usr/local/lib/libcfitsio.dylib (found version "3.390")
-- Looking for dgemm_
-- Looking for dgemm_ - found
-- A library with BLAS API found.
-- Looking for cheev_
-- Looking for cheev_ - found
-- A library with LAPACK API found.
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/local/Cellar/hdf5/1.10.0-patch1/lib/libhdf5.dylib;/usr/local/opt/szip/lib/libsz.dylib;/usr/lib/libz.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib (found version "1.10.0.1")
-- Found the following Casacore components:
--   coordinates
--   images
--   msfits
--   lattices
--   mirlib
--   fits
--   ms
--   measures
--   scimath
--   tables
--   scimath_f
--   casa
-- A library with BLAS API found.
-- A library with LAPACK API found.
-- Found OpenBLAS libraries: /usr/local/opt/openblas/lib/libopenblas.dylib
-- Found OpenBLAS include: /System/Library/Frameworks/vecLib.framework/Headers
-- Checking for module 'glib-2.0'
--   Found glib-2.0, version 2.54.3
--
############################
#   Configuration summary
############################
-- CMAKE_SYSTEM .......... = Darwin-17.4.0
-- CMAKE_BUILD_TYPE ...... =
-- BUILD_SHARED_LIBS ..... =
-- CMAKE_INSTALL_NAME_DIR  =
-- USE_STACKTRACE ........ =
-- CMAKE_CXX_COMPILER .... = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- CMAKE_CXX_FLAGS ....... = -g -O3 -Wall -Wextra -std=c++11 -W -pedantic -ansi -fPIC -fpermissive -fno-omit-frame-pointer -fno-inline-functions -fno-inline-functions-called-once -fno-optimize-sibling-calls
-- ENABLE_MPI ............ = OFF
-- ENABLE_CUDA ........... = Off
-- CASACORE_INC........... = /Users/dijkema/opt/casacore/include
-- CASACORE_LIB........... = /Users/dijkema/opt/casacore/lib/libcasa_coordinates.dylib;/Users/dijkema/opt/casacore/lib/libcasa_images.dylib;/Users/dijkema/opt/casacore/lib/libcasa_msfits.dylib;/Users/dijkema/opt/casacore/lib/libcasa_lattices.dylib;/Users/dijkema/opt/casacore/lib/libcasa_mirlib.dylib;/Users/dijkema/opt/casacore/lib/libcasa_fits.dylib;/Users/dijkema/opt/casacore/lib/libcasa_ms.dylib;/Users/dijkema/opt/casacore/lib/libcasa_measures.dylib;/Users/dijkema/opt/casacore/lib/libcasa_scimath.dylib;/Users/dijkema/opt/casacore/lib/libcasa_tables.dylib;/Users/dijkema/opt/casacore/lib/libcasa_scimath_f.dylib;/System/Library/Frameworks/Accelerate.framework;/System/Library/Frameworks/Accelerate.framework;/Users/dijkema/opt/casacore/lib/libcasa_casa.dylib;/usr/local/Cellar/hdf5/1.10.0-patch1/lib/libhdf5.dylib;/usr/local/opt/szip/lib/libsz.dylib;/usr/lib/libz.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib;/usr/lib/libm.dylib
-- OpenBLAS-LIB .......... = /usr/local/opt/openblas/lib/libopenblas.dylib
-- GLIB-INC .............. = /usr/local/Cellar/glib/2.54.3/include/glib-2.0;/usr/local/Cellar/glib/2.54.3/lib/glib-2.0/include;/usr/local/opt/gettext/include;/usr/local/Cellar/pcre/8.41/include
-- GLIB-LIBS ............. = glib-2.0;intl
-- LAPACK-LIBS ........... = /System/Library/Frameworks/Accelerate.framework;/System/Library/Frameworks/Accelerate.framework
-- CFITSIO-INC  .......... = /usr/local/include
-- CFITSIO-LIBS .......... = /usr/local/lib/libcfitsio.dylib;/usr/lib/libm.dylib
-- WCSLIB-INC  ........... = /usr/local/include
-- WCSLIB-LIBS ........... = /usr/local/lib/libwcs.dylib;/usr/lib/libm.dylib
-- HDF5-INC  ........... = /usr/local/Cellar/hdf5/1.10.0-patch1/include;/usr/include;/usr/local/opt/szip/include
-- HDF5-LIBS ........... = /usr/local/Cellar/hdf5/1.10.0-patch1/lib/libhdf5.dylib;/usr/local/opt/szip/lib/libsz.dylib;/usr/lib/libz.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib
--


Configuring lib/Dirac
-- Compiling lib/Dirac without CUDA support.
-- lib/Dirac objects --> admm_solve;clmfit_nocuda;manifold_average;mdl;myblas;robust_lbfgs_nocuda;robustlm;rtr_solve_robust_admm;updatenu;lmfit_nocuda;consensus_poly;lbfgs_nocuda;rtr_solve;rtr_solve_robust
--


Configuring lib/Radio
-- Compiling lib/Radio without CUDA support.
-- lib/Radio objects --> readsky;predict;residual;transforms;stationbeam;predict_withbeam
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/dijkema/opt/sagecal/sagecal/build
➜  build git:(openblas_root) ✗ make
Scanning dependencies of target rtr_solve_robust
[  1%] Building C object src/lib/Dirac/CMakeFiles/rtr_solve_robust.dir/rtr_solve_robust.c.o
[  3%] Linking C shared library ../../../dist/lib/librtr_solve_robust.dylib
Undefined symbols for architecture x86_64:
  "_my_caxpy", referenced from:
      _fns_R in rtr_solve_robust.c.o
      _itrr in rtr_solve_robust.c.o
      _tcg_solve in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_my_ccopy", referenced from:
      _rtr_solve_nocuda_robust in rtr_solve_robust.c.o
      _armijostep in rtr_solve_robust.c.o
      _fns_R in rtr_solve_robust.c.o
      _itrr in rtr_solve_robust.c.o
      _tcg_solve in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
      _fns_proj in rtr_solve_robust.c.o
      ...
  "_my_cdot", referenced from:
      _fns_g in rtr_solve_robust.c.o
      _fns_proj in rtr_solve_robust.c.o
  "_my_cnrm2", referenced from:
      _itrr in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_my_cscal", referenced from:
      _armijostep in rtr_solve_robust.c.o
      _itrr in rtr_solve_robust.c.o
      _fns_fgrad in rtr_solve_robust.c.o
      _tcg_solve in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_my_dcopy", referenced from:
      _rtr_solve_nocuda_robust in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_my_ddot", referenced from:
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_my_dscal", referenced from:
      _fns_fcount in rtr_solve_robust.c.o
  "_my_idamax", referenced from:
      _fns_fcount in rtr_solve_robust.c.o
  "_my_zgels", referenced from:
      _fns_proj in rtr_solve_robust.c.o
  "_my_zgemm", referenced from:
      _fns_proj in rtr_solve_robust.c.o
  "_setweights", referenced from:
      _rtr_solve_nocuda_robust in rtr_solve_robust.c.o
      _nsd_solve_nocuda_robust in rtr_solve_robust.c.o
  "_update_nu", referenced from:
      _fns_fupdate_weights in rtr_solve_robust.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [dist/lib/librtr_solve_robust.dylib] Error 1
make[1]: *** [src/lib/Dirac/CMakeFiles/rtr_solve_robust.dir/all] Error 2
make: *** [all] Error 2

add install targets for cmake

clang tools scan-build and scan-view detect multiple issues in building libdirac.a for GPU

so this is from scan-view on the output of
scan-build make -f Makefile.gpu
in Dirac

Excluding the dead assignments:

Unix API | Allocator sizeof operand mismatch | lbfgs.c | 228 | 1
Logic error | Assigned value is garbage or undefined | lmfit.c | 457 | 26 |  
Logic error | Assigned value is garbage or undefined | lmfit.c | 773 | 23
Logic error | Assigned value is garbage or undefined | lmfit.c | 1324 | 23
Logic error | Dereference of null pointer | lmfit.c | 1838
Logic error | Dereference of null pointer | admm_solve.c | 915 | 24
Memory Error | Memory leak | lmfit.c | 951 | 33
Logic error | Result of operation is garbage or undefined | lmfit.c | 857 | 25
Logic error | Result of operation is garbage or undefined | lmfit.c | 931 | 27
Logic error | Result of operation is garbage or undefined | lmfit.c | 1958 | 30
Logic error | Result of operation is garbage or undefined | lmfit.c | 2080 | 31
Logic error | Result of operation is garbage or undefined | admm_solve.c | 1000 | 24
Logic error | Result of operation is garbage or undefined | admm_solve.c | 1084 | 25
Logic error | Result of operation is garbage or undefined | admm_solve.c | 1402 | 22 |
Unix API | Undefined allocation of 0 bytes (CERT MEM04-C; CWE-131) | robust_lbfgs_nocuda.c | 162 | 16
Logic error | Uninitialized argument value | clmfit.c | 1011 | 23
Logic error | Uninitialized argument value | robustlm.c | 476 | 26
Logic error | Uninitialized argument value | oslmfit.c | 521 | 36

add OSX to continuous integration

check all the TODO tags in the code

install python scripts in src/buildsky

convert_skymodel.py
create_clusters.py
should also be part of cmake installation

code analysis is needed

it is strange to have the following line in lib/Dirac/Dirac.h
#ifndef SAGECAL_H
#define SAGECAL_H

A code analysis is needed. A simplest analysis can be performed using
http://cppcheck.sourceforge.net/

clang tools scan-build and scan-view detect multiple issues in building libradio.a for GPU

so this is from scan-view on the output of
scan-build make -f Makefile.gpu
in Radio

Excluding the dead assignments:

Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 236 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 240 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 244 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 850 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 854 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 858 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 1409 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 1413 | 1
Unix API | Allocator sizeof operand mismatch | predict_withbeam_gpu.c | 1417 | 1
Logic error | Result of operation is garbage or undefined | readsky.c | 704 | 12
Logic error | Uninitialized argument value | predict_withbeam_gpu.c | 445 | 12
Logic error | Uninitialized argument value | predict_withbeam_gpu.c | 1061 | 12
Logic error | Uninitialized argument value | predict_withbeam_gpu.c | 1630 | 19

Build problem in gpu_support branch: "_GLIBCXX_MATH_H" is not defined

Describe the bug
From the build directory in the gpu_support branch:
make
gives
from <command-line>:0: /cm/shared/package/cuda91/toolkit/9.1.85/include/crt/math_functions.h:8891:5: error: "_GLIBCXX_MATH_H" is not defined [-Werror=undef] #if _GLIBCXX_MATH_H && __cplusplus >= 201103L ^

To Reproduce
Steps to reproduce the behavior:

1. cd build
2. module load cuda91
3. module load cmake/3.8.2
4. module load casacore/2.3.0-gcc-4.9.3
5. cmake .. -DCMAKE_INSTALL_PREFIX=/home/hspreeuw/sagecal-code/src/MS -DHAVE_CUDA=TRUE
6. make

Expected behavior
A complete build

short installation guide for OSX might be necassary

Changelog should be updated

Installation instructions must be updated

scan-build detects a few bugs in building sagecal executable for CPU

so from scan-build make from within MS:

CPU and GPU version should be separated

It will be a lot easier if we have GPU and CPU versions in separate repositories.

Make threadfn_fns_fcount faster

threadfn_fns_fcount takes 23.59% of the total run time with 40 threads (-n 40) on a CPU with 40 logical cores.
With -n 1 this is only 0.83%.
But -n 1 is, of course, not desirable for the total run time of Sagecal.

Since threadfn_fns_fcount is a very simple function, it should be possible to have it take a negligible fraction of the total runtime of Sagecal on any CPU.

CITATION.cff errors

1. According to the spec and schema, orcid ids are in fact strings that include the https://orcid.org part.
   • https://citation-file-format.github.io/assets/pdf/cff-specifications-1.0.3.pdf
   • https://github.com/citation-file-format/schema/blob/master/CFF-Core/schema.yaml.
2. Ben's last name should be split into a family-names "Werkhoven" and a name-particle "van" (note lowercase)

Check Rdds

Check if the use of RDDs is feasible.

https://spark.apache.org/docs/latest/rdd-programming-guide.html

Update contact text

The contact text in executable help should be updated.
Currently we have
Report bugs to <[email protected]>
It should point to our Github Repo.

unit testing is needed

a guide for contributers is needed

add CONTRIBUTING.md

improve documentation

documentation needs to be more structured and clean
For example:
move info to relevant sections
create separate markdown files if necessary
...

add MPI test to Travis Docker images

MPI packages should be installed for Docker images and mpi build should be tested

Test all the builds with newer version of gcc

All the builds work with GCC 4.9.3

More tests with newer version of GCC is needed.

Fix memory leaks detected by cppcheck.

Lots of memory leaks detected by cppcheck, but also other issues.
See attached cppcheck output .
cppcheck.txt

add GPU support to Cmake build system

continues integration must be added

Test directory is too big

The code itself is ~3mb but the size of the files in test directory are about 55 mb. Maybe we can store test and example data in a separate repository.

Kernsuite Debian package for sagecal latest release

make[4]: Leaving directory '/build/sagecal-0.5.0/obj-x86_64-linux-gnu'
make -f lib/Dirac/CMakeFiles/dirac.dir/build.make lib/Dirac/CMakeFiles/dirac.dir/build
make[4]: Entering directory '/build/sagecal-0.5.0/obj-x86_64-linux-gnu'
[  1%] Building C object lib/Dirac/CMakeFiles/robustlm.dir/robustlm.o
cd /build/sagecal-0.5.0/obj-x86_64-linux-gnu/lib/Dirac && /usr/bin/cc -Drobustlm_EXPORTS -I/build/sagecal-0.5.0/src/lib/Dirac/.  -g -O2 -fdebug-prefix-map=/build/sagecal-0.5.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -o CMakeFiles/robustlm.dir/robustlm.o   -c 
/build/sagecal-0.5.0/src/lib/Dirac/robustlm.c
In file included from /build/sagecal-0.5.0/src/lib/Dirac/Dirac.h:26:0,
                 from /build/sagecal-0.5.0/src/lib/Dirac/robustlm.c:27:
/usr/include/glib-2.0/glib.h:30:10: fatal error: glib/galloca.h: No such file or directory
 #include <glib/galloca.h>
          ^~~~~~~~~~~~~~~~
compilation terminated.
lib/Dirac/CMakeFiles/robustlm.dir/build.make:65: recipe for target 'lib/Dirac/CMakeFiles/robustlm.dir/robustlm.o' failed```

add mpi , restore and buildsky options to cmake build

add proper software test

CI should run examples to test the software. Also related to #72

a better build system is needed

alternatives:
cmake
meson
jinja

a short tutorial can be added

it would be very useful to have a short tutorial which explains how to use the code. We have an example but it could be more explanatory.

DOI needs to be added