Replace `up_solve` with petsc KSP

[kvslab]

Replace `up_solve` with petsc KSP

[keiyamamo]

Hi @jorgensd @johannesring @dbruneau-mie

I was investigating the error and found something we could try. It might not work, but I thought it’s worth mentioning here.
So the error message we get is

Error:   Unable to solve linear system using PETSc Krylov solver.
Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
 Where:   This error was encountered inside PETScKrylovSolver.cpp.

where we use mumps as linear solver.
What I found online is

PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00
They are caused by MUMPS and the best chance to avoid them is to set the MOD class keyword argument serial_ordering to True or slightly change the number of mpirun processes for the simulation of a specific setup.

from https://custem.readthedocs.io/en/latest/tipsandtricks.html
Looking at their source code, I figure out that by setting serial_ordering to True in their class means

- serial_ordering = False, type bool
            use serial ordering of *METIS* instead parallel ordering of
            *SCOTCH*; leads to slight increase of solution time for *MUMPS*
            but significantly reduces chance of potential *MUMPS* crashes

import dolfin as df
if serial_ordering:
   df.PETScOptions.set("mat_mumps_icntl_28", 1)

where petsc document says

-mat_mumps_icntl_28 - ICNTL(28): use 1 for sequential analysis and ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering

My guess is that we have to use

parameters["mesh_partitioner"] = "ParMETIS"

and also use PETScOptions. At the moment, I do not know how to do it immediately because I do not have much experience with petsc but it should not be too hard...
Please let me know if you have any suggestion or comment.

[keiyamamo]

For now, I concluded that adding

PETScOptions.set("mat_mumps_icntl_14", 400) # allocate more memory to mumps 

inside newtonsolver.py may reduce the possibility of the error.