Two problems at the same time

[juliasmoothoptimizers]

Two problems at the same time

[dpo]

Actually, it depends what kind of parallelism you have in mind. If you use a queuing system such as Sungrid Engine or LSF to dispatch problems on different, independent machines, and if a fresh CUTEst module is initialized independently on each machine, you should be ok.

[timholy]

Interestingly, even with #97 this doesn't work:

julia> nlp1 = CUTEstModel("HS35")
Minimization problem HS35
nvar = 3, ncon = 1 (1 linear)


julia> nlp2 = CUTEstModel("BARD")
ERROR: CUTEst.CUTEstException(1,"memory allocation error")
 in #CUTEstModel#13(::Bool, ::Bool, ::Type{T}, ::String) at /home/tim/.julia/v0.5/CUTEst/src/CUTEst.jl:138
 in CUTEst.CUTEstModel(::String) at /home/tim/.julia/v0.5/CUTEst/src/CUTEst.jl:99

julia> finalize(nlp1)

julia> nlp2 = CUTEstModel("BARD")
Minimization problem BARD
nvar = 3, ncon = 0 (0 linear)

That line corresponds to the call to :fortran_open_, so it's generated by the library rather than julia.

I think that's a strong argument against #97.

[abelsiqueira]

Actually, I think the problem is in usetup.

It seems the problem is the use of global variables inside fortran CUTEst. I don't think there is any interest in changing that, so maybe this should be closed as wontfix? @dpo, what do you think?

[anriseth]

I was hoping to run lots of CUTEst problems in parallel for some benchmarking. Is that possible currently / are there any hopes of enabling that?

[abelsiqueira]

Currently not possible, and maybe impossible without changing the CUTEst source.
I have updated the branch with non-global cutest_lib in case you want to try to work this out. The example above fails in the same way in the new branch.

[anriseth]

Actually, it depends what kind of parallelism you have in mind.

I only have access to four independent machines, so I wanted to use the 32 cores on a single machine rather than going parallel between machines.

This is basically what I had in mind originally:

addprocs(2)
@everywhere using CUTEst
@everywhere function opt(name)
  nlp = CUTEstModel(name)
  retval = obj(nlp,nlp.meta.x0)
  finalize(nlp)
  retval
end

pmap(opt, CUTEst.select(contype="unc"))

On both th-multi and master:

ld: cannot find ELFUN.o: No such file or directory
ld: cannot find EXTER.o: No such file or directory
ld: cannot find GROUP.o: No such file or directory
ld: cannot find RANGE.o: No such file or directory
ERROR: On worker 3:
failed process: Process(`ld -shared -o libHILBERTA.so ELFUN.o EXTER.o GROUP.o RANGE.o /scratch/riseth/julia-pkgs/CUTEst/deps/files/cutest/objects/pc64.lnx.gfo/double/libcutest_double.so /home/riseth/.julia/v0.6/CUTEst/src/../deps/usr/lib/libgfortran.so`, ProcessExited(1)) [1]
pipeline_error at ./process.jl:682
run at ./process.jl:651
#13 at /home/riseth/.julia/v0.6/CUTEst/src/CUTEst.jl:98
cd at ./file.jl:70
#sifdecoder#12 at /home/riseth/.julia/v0.6/CUTEst/src/CUTEst.jl:92
#sifdecoder at ./<missing>:0
#16 at /home/riseth/.julia/v0.6/CUTEst/src/CUTEst.jl:119
cd at ./file.jl:70
#CUTEstModel#15 at /home/riseth/.julia/v0.6/CUTEst/src/CUTEst.jl:111
opt at ./REPL[7]:2
#106 at ./distributed/process_messages.jl:268 [inlined]
run_work_thunk at ./distributed/process_messages.jl:56
macro expansion at ./distributed/process_messages.jl:268 [inlined]
#105 at ./event.jl:73
Stacktrace:
 [1] #571 at ./asyncmap.jl:178 [inlined]
 [2] foreach(::Base.##571#573, ::Array{Any,1}) at ./abstractarray.jl:1733
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Base.Distributed.##204#207{WorkerPool}, ::Function, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at ./asyncmap.jl:154
 [5] #async_usemap#556(::Function, ::Void, ::Function, ::Base.Distributed.##188#190, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at ./asyncmap.jl:103
 [6] (::Base.#kw##async_usemap)(::Array{Any,1}, ::Base.#async_usemap, ::Function, ::Array{Any,1}, ::Vararg{Array{Any,1},N} where N) at ./<missing>:0
 [7] (::Base.#kw##asyncmap)(::Array{Any,1}, ::Base.#asyncmap, ::Function, ::Array{Any,1}) at ./<missing>:0
 [8] #pmap#203(::Bool, ::Int64, ::Void, ::Array{Any,1}, ::Void, ::Function, ::WorkerPool, ::Function, ::Array{Any,1}) at ./distributed/pmap.jl:126
 [9] pmap(::WorkerPool, ::Function, ::Array{Any,1}) at ./distributed/pmap.jl:101
 [10] #pmap#213(::Array{Any,1}, ::Function, ::Function, ::Array{Any,1}) at ./distributed/pmap.jl:156
 [11] pmap(::Function, ::Array{Any,1}) at ./distributed/pmap.jl:156

[anriseth]

Okay, so I can fix the issue with missing ELFUN.o etc. by using decode=false. But then I seem to fall into the same problem as @timholy mentioned with :fortran_open_.

UPDATE: The function data is stored in OUTSDIF.d generated by sifdecoder, which the Fortran libraries call. So we would need a way to tell the library to have separate OUTSDIF.d files for each problem.

OLD:
This means that the functionality with decode=false does not work (as I expected it to):

julia> nlp = CUTEstModel("AKIVA")
Minimization problem AKIVA
nvar = 2, ncon = 0 (0 linear)
julia> finalize(nlp)
julia> nlp = CUTEstModel("BROWNDEN")
Minimization problem BROWNDEN
nvar = 4, ncon = 0 (0 linear)
julia> finalize(nlp)
julia> nlp = CUTEstModel("AKIVA";decode=false)
Minimization problem AKIVA
nvar = 4, ncon = 0 (0 linear)

As you can see, AKIVA should have nvar = 2, however, it gets the values of BROWNDEN if decode=false.