Parallel running gets stuck with the following error message

[GitPaean]

Parallel running gets stuck with the following error message

[blattms]

To be able to fix we need a backtrace to see at what stage this happening or a test case to reproduce.

[GitPaean]

With Norne and 128 mpi processes, there are three processes complaining this error and the whole simulation will get stuck.

It is understandable with more processes, the domain decomposition is harder, but the symptom is something needs to be fixed.

[blattms]

I would need a new computer for replicating this. On mine MPI segfaults during initialization with 128 processes. Or access to one.

In the meantime please checkout #483, compile it with debugging symbols, run and post the output. It needs gdb and should print the loadbalancing information and a backtrace.

[blattms]

I produces a synthetic test case.

The problem is that the only explanation is that one process ends up with no cells. There is no reason SparseTabel should have problems with zeros rows. Unfortunately, allowing this reveals more bugs (e.g. the region stuff used by the RateConverter is broken
and only runs through if all processes have cells in the same amount of regions, even then it might be wrong in parallel). Hence this will take some time...

BTW it seems a bit surprising that Zoltan would create partitions with 0 cells for Norne for 128 processes (thats lousy). Please run the branch to verify that and tells us how Zoltan is configured/built.

[GitPaean]

. Please run the branch to verify that and tells us how Zoltan is configured/built.

I tried and then I saw the problem related to UMFPack. Let me fix that first.

[GitPaean]

Do you mean PR #483 or #484? or both.

[GitPaean]

I sent you the output to your email, @blattms . I think I used the email with dr-blatt.de.

[blattms]

Thanks that is sufficient. I supports my assumption that some ranks end up without any cells (which means the Zoltan loadbalancer performs a bit poorly).

Did you compile Zoltan yourself, too? How did you configure it? Does it use PTScotch?

[GitPaean]

I compiled Zoltan from the Trilinos-master, it did not use PTScotch.

 cmake ../ -DCMAKE_INSTALL_PREFIX=${install_fix} -DTPL_ENABLE_MPI:BOOL=ON -DTrilinos_ENABLE_Zoltan:BOOL=ON -DTrilinos_ENABLE_Zoltan:BOOL=ON -DCMAKE_BUILD_TYPE=Release -Wno-dev

[blattms]

Unfortunately, the attempt to get a stack trace was not successful. I guess the cluster did not allow for spawning gdb as intended by #483 .

If you have time it woud be nice if you could double check whether #484 exits gracefully for you.

I guess you also want to try to use ZOLTAN a supported external loadbalancer like PTScotch or another one. Hopefully that will do a better job.

[blattms]

Once I fix #484 that is. Jenkins complains.

[GitPaean]

If you have time it woud be nice if you could double check whether #484 exits gracefully for you.

I guess you also want to try to use ZOLTAN a supported external loadbalancer like PTScotch or another one. Hopefully that will do a better job.

I will test both. But I need to figure out the compilation related to Suitesparse first.