juliasmoothoptimizers/cutest.jl

Julia's CUTEst Interface

cutest.jl's Issues

Use AbstractNLPModel

For compatibility with NLPModels.jl and Optimize.jl.
See this

Documentation

Just a thought. I wonder if the documentation for the core interface functions could be the contents of the corresponding man page. For example help(usetup) would display the result of man cutest_usetup. Is that possible?

Full compatibility with AbstractModel

After #53

Create select that returns a generator expression

Equivalent to
(CUTEstModel(p) for p in CUTEst.select(...))

Strange values for HIELOW

Hey,

I have been using CUTEst.jl and noticed that the "HIELOW" problem (and some other hierarchical logit models) seem to have unbounded objectives. For example, with "HIELOW" if i set
x = [0.0, 0.0, 1e20]

I get an objective value of:
-5.746190126842027e22

Which doesn't really make sense (the problem is min -log like).

NLP.jl missing

I can't find NLP.jl necessary to run this package. The links in the README no longer work and I can't find it on google.

Thank you for providing the link.

Simplify?

CUTEst has too many methods. Besides the core functions and the NLPModels variant, there is also the specialized interface, which I'm not completely sure how useful it is.

For each core function, the specialized interface has between one and six versions of that function.

• One returning all outputs
• One inplace version when there are arrays in the output
• One additional inplace version when there are integer arrays in the output (Int->Cint)
• If some inputs can be replaced by nlp, then another version for each of the previous ones.

See ugh, for an example using all six.

Do we need all these versions?

I think a leaner solution would be creating only two interfaces

• Both using nlp
• One returning all outputs
• One inplace

Interface with MathProgBase

We should do it for compatibility. The AMPL interface can serve as a model. Seems pretty straightforward.

Manually updating the specialized function - end of the script era

I started making updates on the specialized functions, without the scripts. Some fine-tuning is required, and the script was becoming too exception-based.

That said, if there is some big change to do, it is better to do now.

Use deps/build.jl

Suggested by @timholy.

Calling two different problems, one after the other, breaks the second.

At least as far as objective value in concerned.
In https://travis-ci.org/abelsiqueira/CUTEst.jl/jobs/50675367, first we decode HS35 and print the objective value correctly. Then, we decode HS53, and obtain a incorrect value for the objective value.
Things worth noting:

• If I change the order of tests called in runtests.jl (first and second line), then HS53 is correct and HS35 is not.
• Using the libname as the problem name (the default way of calling CUTEstModel), doesn't change anything.
• If I change HS53 to HS31, it appears to not have a problem. Since HS35 and HS31 have the same dimensions, this appears to be related to memory allocation.
• As above, if I change HS35 to HS51 or HS52, is appears to work, and again they have the same dimensions.
• Same for HS32 and HS33.

This leads me to think that this is a problem with CUTEst in Fortran. It appears to use something in the allocation process that it keeps in the memory even after terminate is called. Unfortunately, the code is beyond me.

Moving to new organization

@abelsiqueira I'm about to move this repository to another organization of mine. It shouldn't affect you much. It should be sufficient to add the new repo address as an upstream.

Strange error with `ushprod`

Was #126.

I updated the example to check all problems with two variables or less, and many of them have errors.
However, when I change the code to not allocate the input so many times, it works normally.

The failing code is here, and I think it may be simplified.

The change to make it work is simply:

ushprod(io_err, Cint[n], goth, x0, Cint[1], Cint[1], Cdouble[1.0], nnzr, Ir, r)
ushprod(io_err, Cint[n], goth, x0, Cint[1], Cint[1], Cdouble[1.0], nnzr2, Ir2, r2)

to

ushprod(io_err, Cint[n], goth, x0, nnzs, Is, s, nnzr, Ir, r)
ushprod(io_err, Cint[n], goth, x0, nnzs, Is, s, nnzr2, Ir2, r2)

This makes me think it's something related to memory allocation. I'm not sure what, though, or how to prevent this.

Release checklist

• Fix the specialized interface (#38)
• Generate the documentation
  • Core and specialized interface (#32)
  • Update the README.md
  • Make an read the docs online documentation.
• #105
• #107
• Change develop to master
• Decide the version number: v0.1.0

What else we should do to release a stable version?

Warnings

I'm getting the warnings

WARNING: Method definition cchprods!(Int64, Int64, Bool, Array{Float64, 1}, Array{Float64, 1}, Int64, Array{Float64, 1}, Array{Int64, 1}, Array{Int64, 1}) in module CUTEst at /Users/dpo/.julia/v0.5/CUTEst/src/specialized_interface.jl:5294 overwritten at /Users/dpo/.julia/v0.5/CUTEst/src/specialized_interface.jl:5304.
WARNING: Method definition cchprods!(CUTEst.CUTEstModel, Bool, Array{Float64, 1}, Array{Float64, 1}, Int64, Array{Float64, 1}, Array{Int64, 1}, Array{Int64, 1}) in module CUTEst at /Users/dpo/.julia/v0.5/CUTEst/src/specialized_interface.jl:5351 overwritten at /Users/dpo/.julia/v0.5/CUTEst/src/specialized_interface.jl:5363.

Also

WARNING: Method definition f(Any) in module Main at /Users/dpo/.julia/v0.5/CUTEst/test/hs32.jl:4 overwritten at /Users/dpo/.julia/v0.5/CUTEst/test/hs4.jl:4.
WARNING: Method definition g(Any) in module Main at /Users/dpo/.julia/v0.5/CUTEst/test/hs32.jl:5 overwritten at /Users/dpo/.julia/v0.5/CUTEst/test/hs4.jl:5.
WARNING: Method definition H(Any) in module Main at /Users/dpo/.julia/v0.5/CUTEst/test/hs32.jl:8 overwritten at /Users/dpo/.julia/v0.5/CUTEst/test/hs4.jl:6.
WARNING: Method definition #H(Array{Any, 1}, Main.#H, Any) in module Main overwritten.

We should probably give each problem function names other than just f, g and H.

Make `Pkg.build` reinstall cutest/sifdecode/mastsif

Better would be check if a reinstall is needed, but this would be OS specific.

Related to #128.

UndefVarError: NLPModelEvaluator not defined

julia> using CUTEst
INFO: Recompiling stale cache file /Users/dpo/.julia/lib/v0.6/CUTEst.ji for module CUTEst.
ERROR: LoadError: LoadError: UndefVarError: NLPModelEvaluator not defined
Stacktrace:
 [1] include_from_node1(::String) at ./loading.jl:576
 [2] include(::String) at ./sysimg.jl:14
 [3] include_from_node1(::String) at ./loading.jl:576
 [4] include(::String) at ./sysimg.jl:14
 [5] anonymous at ./<missing>:2
while loading /Users/dpo/.julia/v0.6/CUTEst/src/mpb_interface.jl, in expression starting on line 6
while loading /Users/dpo/.julia/v0.6/CUTEst/src/CUTEst.jl, in expression starting on line 72

This must be related to factoring out NLPJuMPModel.

KOWOSBNE is "unc", but has ncon = 11?

I got "KOWOSBNE" from CUTEst.select(contype="unc"), however, nlp.meta.ncon = 11.

Is this a mistake in MASTSIF that I should report, or is there a reason for this?

New release

Current one doesn't work with 0.6.

Increase coverage

I hope to achieve 90%.

Linuxbrew

Linuxbrew has been picking up steam. Recently, I forked Homebrew.jl and made Linuxbrew.jl. Some tests are currently failing and I'm not yet sure why but it seems inconsequential. I can install formulae, including precompiled bottles.

I just installed and tested CUTEst with Linuxbrew and all works well. I also tested the installation within Julia using this modified build.jl: https://gist.github.com/3de7180903daf6af3004cc50b119864d

Lo and behold: it works!

Pkg.test("CUTEst") is a drag because dependencies like IPOPT build lots of redundant stuff, but hopefully, that can be improved.

It would be great if you could give it a try.

Use native CUTEst and gcc if installed

Improve deps/build.jl to reuse CUTEst and gcc if they are already installed.
See MUMPS.jl for example.

Page build failure

Documentation build needs update.

SCURLY10: wrong starting point

For me on OSX with Julia 0.6, CUTEstModel("SCURLY10") reports a zero starting point, which is not what's indicated in the SIF file. A Fortran driver reports the correct (nonzero) starting point.

Should `hess` return only the upper triangular part?

The function hess from the julia_interface.jl is currently returning a sparse upper triangular matrix, because it is symmetric. But since this interface should be easier to use, wouldn't it be better to return a matrx with both sides filled? This will (almost) double allocation, but will enable very high level code, like

hess(nlp, x)\v

Finite Element Format

The CUTE paper says that the format is defined in this book, which I don't have. @dpo, do you have it?
Otherwise, I'm gonna write some trivial function calls just to increase coverage.

Missing files

I built the branch from #121 but there I have some problems selected by

unc_problems = CUTEst.select(max_var=10,contype = :unc, custom_filter=x->x["derivative_order"]>=2)

that gives me this error

gfortran: error: ELFUN.f: No such file or directory
gfortran: error: EXTER.f: No such file or directory
gfortran: error: GROUP.f: No such file or directory
gfortran: error: RANGE.f: No such file or directory
gfortran: fatal error: no input files
compilation terminated.
ERROR: LoadError: failed process: Process(`gfortran -c -fPIC ELFUN.f EXTER.f GROUP.f RANGE.f`, ProcessExited(1)) [1]
Stacktrace:
 [1] pipeline_error(::Base.Process) at ./process.jl:687
 [2] run(::Cmd) at ./process.jl:656
 [3] (::CUTEst.##23#24{Bool,String,Tuple{},String,String,String})() at /home/pkm/.julia/v0.6/CUTEst/src/CUTEst.jl:97
 [4] cd(::CUTEst.##23#24{Bool,String,Tuple{},String,String,String}, ::String) at ./file.jl:70
 [5] #sifdecoder#22(::Bool, ::Function, ::String) at /home/pkm/.julia/v0.6/CUTEst/src/CUTEst.jl:92
 [6] (::CUTEst.#kw##sifdecoder)(::Array{Any,1}, ::CUTEst.#sifdecoder, ::String) at ./<missing>:0
 [7] (::CUTEst.##26#28{Bool,Bool,String,Tuple{}})() at /home/pkm/.julia/v0.6/CUTEst/src/CUTEst.jl:119
 [8] cd(::CUTEst.##26#28{Bool,Bool,String,Tuple{}}, ::String) at ./file.jl:70
 [9] #CUTEstModel#25(::Bool, ::Bool, ::Type{T} where T, ::String) at /home/pkm/.julia/v0.6/CUTEst/src/CUTEst.jl:111
 [10] CUTEst.CUTEstModel(::String) at /home/pkm/.julia/v0.6/CUTEst/src/CUTEst.jl:107
 [11] macro expansion at /home/pkm/.julia/v0.6/OptimTests/test/runtests.jl:36 [inlined]
 [12] anonymous at ./<missing>:?
 [13] include_from_node1(::String) at ./loading.jl:539
 [14] include(::String) at ./sysimg.jl:14
 [15] process_options(::Base.JLOptions) at ./client.jl:305
 [16] _start() at ./client.jl:371
while loading /home/pkm/.julia/v0.6/OptimTests/test/runtests.jl, in expression starting on line 32

the offending problems are the ones that end with "LS" plus KOWOSBNE and SSI.

Error in cons_coord

julia> using CUTEst

julia> model = CUTEstModel("BA-L16");  # be patient

julia> c, J = consjac(model, model.meta.x0);
ERROR: CUTEst.CUTEstException(2, "array bound error")
Stacktrace:
 [1] cons_coord(::CUTEst.CUTEstModel, ::Array{Float64,1}) at /Users/dpo/.julia/v0.6/CUTEst/src/julia_interface.jl:128
 [2] consjac(::CUTEst.CUTEstModel, ::Array{Float64,1}) at /Users/dpo/.julia/v0.6/CUTEst/src/julia_interface.jl:149

Enable attobot

@dpo
https://github.com/integration/attobot

Segmentation fault

@dpo, I had a weird segmentation fault (random occurrence), could you try to recreate?

using CUTEst
nlp = CUTEstModel("ROSENBR")
for i = 1:100000
    uhprod(nlp, false, [0.0;0.0], [0.0;0.0])
end

Store files somewhere beside /tmp?

Every time I reboot my laptop (e.g., after security updates that affect the kernel) I need to do Pkg.rebuild("CUTEst"). Any thoughts on storing the files elsewhere?

Cap on NLPModels?

Why is there an upper bound on the version of NLPModels in v0.3.5?

Pros and cons of using other interfaces inside `julia_interface.jl`

We need to evaluate the pros and cons of using the core or specialized interfaces inside julia_interface.jl.

From #40:

The call chain would be

• objcons(nlp, x) in the Julia interface
• ufn(nlp, x) in the specialized interface
• ufn(io_err, n, x, f, libname) in the core interface
• cutest_ufn() in libcutest.

Right now, objcons calls cutest_ufn directly. But the intermediate functions are just passing scalars
and pointers around, there's no copy of arrays.

Specialized interface

There's stuff I must have missed in the specialized interface that perplexes me:

There seems to be a ! version of every function. But I can't make sense of some of them. Normally, ! is used when it modifies (one of) its input arguments in place. The convention is also that that argument be returned by the function. (I hear the convention is also that the modified argument should appear first.) When several arguments are modified, I'm not sure what should be done, but in our case, I'm happy with what you've done.

Here are a few cases that confuse me:

1. The following functions don't modify their input arguments. I don't see a need for them.
   • udimen!(input::Int, libname),
   • udimsh!(libname),
   • udimse!(libname)
     *
2. The following functions take a CUTEstModel as input.
   • udimen(nlp::CUTEstModel, input::Int) and udimen!(nlp::CUTEstModel, input::Int)
   • udimsh(nlp::CUTEstModel) and udimsh!(nlp::CUTEstModel)
   • etc.
   • and the same for the constrained functions.

A lot of what those functions do has already happened in the constructor, when the NLPModelMeta field was filled out. If anything is missing (e.g., the variable names), then I think it should go into the constructor. It's ok to add fields to NLPModelMeta if any are missing.

Exceptions are ureport/creport and ustats/cstats, but I wonder if the counting should take place in objgrad, cons, etc. I'm not sure yet what the appropriate interface should be, though. I'm not sure.

I hadn't realized some of the functions in there take a CUTEstModel as input. It feels strange, but I guess it could be useful if you want to be more surgical.

Question: What do we do with the sparse matrices in finite-element format? Is there a package out there that could help us with them?

Documentation

• document all methods (some docstrings from NLPModels could perhaps be reused here?)
  • NLPModels #75
  • specialized and core #76
• write a simple tutorial

fixedlibname as default

The sifdecoder is not using the fixedlibname, so if using the default libname, the call fails.

Update docs

Consider MetaPkg

MetaPkg by tbreloff.

• Is it better than what we do?
• Is it completely enough to do what we do?
• Will community adopt it?

Multipliers in `hess` and `hess_coord` are keyword arguments, which default to `y0`.

I think that the multipliers in hess and hess_coord should not be keyword argument, because hess(nlp, x, y) seems more natural than hess(nlp, x, y=y).

Also, I think that hess(nlp, x) defaulting to hess(nlp, x, y0) is error prone. We should default y to zero, or not use a default and create hessObj and hessLagr separately.

What do you think?

Bug on cdimsh

Some weird bug on cdimsh or on HS10. Possibly on general CUTEst, not sure.

using CUTEst
nlp = CUTEstModel("HS10")
hess(nlp, ones(2), y=[1.0]) # works
cdimsh()
hess(nlp, ones(2), y=[1.0]) # returns a different matrix

HS10 has a linear objective function, but maybe not related.

Test on OSX

Travis-CI has enabled OSX support for our repository. Currently testing in the osx branch.

Segfaults [ON HOLD]

Edit: On hold... So apparently Pkg.update is really giving me a hard time today. JuMP is suddenly version 0.11.3, and the segfaults happened because it took me back to master from develop. Now I can't run anything... I'll close if I resolve it.

So, I updated my CUTEst.jl and now I am getting segfaults

using CUTEst, Optim
for i = 1:100
    nlp=CUTEstModel("CLIFF")
    optimize(x->obj(nlp,x), nlp.meta.x0)
    cutest_finalize(nlp)
end

signal (11): Segmentation fault
cutest_uofg_threadsafe_ at /home/pkm/.linuxbrew/lib/libcutest_double.so (unknown line)
cutest_uofg_ at /home/pkm/.linuxbrew/lib/libcutest_double.so (unknown line)
anonymous at no file:0
unknown function (ip: 0x7fc998d609db)
jl_toplevel_eval_in at /home/pkm/julia/usr/bin/../lib/libjulia.so (unknown line)
objgrad at /home/pkm/.julia/v0.4/CUTEst/src/julia_interface.jl:37
obj at /home/pkm/.julia/v0.4/CUTEst/src/julia_interface.jl:45
jl_apply_generic at /home/pkm/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at none:3
optimize at /home/pkm/.julia/v0.4/Optim/src/nelder_mead.jl:216
jlcall_optimize_21747 at  (unknown line)
jl_apply_generic at /home/pkm/julia/usr/bin/../lib/libjulia.so (unknown line)
optimize at /home/pkm/.julia/v0.4/Optim/src/optimize.jl:25
anonymous at none:3
unknown function (ip: 0x7fc998d609db)
jl_toplevel_eval_in at /home/pkm/julia/usr/bin/../lib/libjulia.so (unknown line)
eval_user_input at REPL.jl:62
jlcall_eval_user_input_21445 at  (unknown line)
jl_apply_generic at /home/pkm/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at REPL.jl:92
unknown function (ip: 0x7fc998d51f53)
unknown function (ip: (nil))
Segmentation fault (core dumped)

Can you replicate this?

How to pass libname to the raw interface

@dpo, the raw interface, by default, does not require the user to pass the library, nor the problem.
That is, he would call

cfn(status, n, m, x, f, c)

Currently this is not possible. We need to decide to either

• pass the problem too, changing the default expected interface; or
• choose a default the libname, (libCUTEstJL, for instance).

Segmentation fault on `cutest_cfn_`

Tried testing the package, but got a segmentation fault.
Downloaded NPL, and did

julia> Pkg.clone(pwd())
julia> Pkg.build("NLP.jl")

Download CUTEst.jl and did inside of folder test

$ LD_LIBRARY_PATH=. julia
julia> include("../src/CUTEst.jl")
julia> include("test_cutest.jl")

Output: https://gist.github.com/abelsiqueira/211d9df07e61926671e7

Implement number of function evaluations inside CUTEstModel

`finalizer`

Add a finalizer for CUTEst.

Related: JuliaSmoothOptimizers/SolverTools.jl#32

Two problems at the same time

Is it possible to open two problems at the same time?

Something in the lines of

nlp1 = CUTEstModel("HS35")
nlp2 = CUTEstModel("BARD")
f1 = obj(nlp1, nlp1.meta.x0)
f2 = obj(nlp2, nlp2.meta.x0)

`jac_op`

jac_op, the Jacobian matrix operator.
Easy, but let's wait on JuliaSmoothOptimizers/LinearOperators.jl/pull/18, too see if something changes.

Zenodo release?

CUTEstModel("GUASS2") throws error

Hey,

When I type:

nlp_raw = CUTEstModel("GAUSS2")

I get an error. I think the problem "GAUSS2" has been added to the latest push of CUTEst because didn't have it in the test set before.

-------- FULL ERROR ---------
gfortran: error: ELFUN.f: No such file or directory
gfortran: error: EXTER.f: No such file or directory
gfortran: error: GROUP.f: No such file or directory
gfortran: error: RANGE.f: No such file or directory
gfortran: fatal error: no input files
compilation terminated.
ERROR: failed process: Process(`gfortran -c -fPIC ELFUN.f EXTER.f GROUP.f RANGE.f`, ProcessExited(1)) [1]
 in pipeline_error(::Base.Process) at ./process.jl:616
 in run at ./process.jl:592 [inlined]
 in (::CUTEst.##23#24{Bool,String,Tuple{}})() at /home/ohinder/.julia/v0.5/CUTEst/src/CUTEst.jl:97
 in cd(::CUTEst.##23#24{Bool,String,Tuple{}}, ::String) at ./file.jl:59
 in #sifdecoder#22(::Bool, ::Function, ::String) at /home/ohinder/.julia/v0.5/CUTEst/src/CUTEst.jl:92
 in (::CUTEst.#kw##sifdecoder)(::Array{Any,1}, ::CUTEst.#sifdecoder, ::String) at ./<missing>:0
 in (::CUTEst.##26#27{Bool,Bool,String,Tuple{}})() at /home/ohinder/.julia/v0.5/CUTEst/src/CUTEst.jl:119
 in cd(::CUTEst.##26#27{Bool,Bool,String,Tuple{}}, ::String) at ./file.jl:59
 in #CUTEstModel#25(::Bool, ::Bool, ::Type{T}, ::String) at /home/ohinder/.julia/v0.5/CUTEst/src/CUTEst.jl:111
 in CUTEst.CUTEstModel(::String) at /home/ohinder/.julia/v0.5/CUTEst/src/CUTEst.jl:107