[Michael Rath]

Dear forum users,

I just realized, that updating to the newest CentOS kernel version 3.10.0-862.11.6 breaks the MPI implementation nearly completely.

I get the following messages in 19.1 with the different run modes:

Intel MPI Local Parallel
works.

IBM MPI Local Parallel
doesn't work:

solver-mpi.exe: Rank 0:0: MPI_Init_thread: ibv_modify_qp(2rtr) failed
solver-mpi.exe: Rank 0:0: MPI_Init_thread: ibv_modify_qpstate() failed
solver-mpi.exe: Rank 0:0: MPI_Init_thread: Internal Error: Processes cannot connect to rdma device
solver-mpi.exe: Rank 0:1: MPI_Init_thread: ibv_modify_qp(2rtr) failed
solver-mpi.exe: Rank 0:1: MPI_Init_thread: ibv_modify_qpstate() failed
solver-mpi.exe: Rank 0:1: MPI_Init_thread: Internal Error: Processes cannot connect to rdma device
MPI Application rank 0 exited before MPI_Finalize() with status 1
An error has occurred in cfx5solve:

The ANSYS CFX solver exited with return code 1.   No results file has been
created.

Intel MPI Distributed Parallel:
doesn't work:

/etc/tmi.conf: No such file or directory
/etc/tmi.conf: No such file or directory
/etc/tmi.conf: No such file or directory
/etc/tmi.conf: No such file or directory
[../../src/mpid/ch3/channels/nemesis/netmod/ofa/ofa_init.c:2099] error(22): Could not modify boot qp to RTR
[../../src/mpid/ch3/channels/nemesis/netmod/ofa/ofa_init.c:2099] error(22): Could not modify boot qp to RTR
[../../src/mpid/ch3/channels/nemesis/netmod/ofa/ofa_init.c:2099] error(22): Could not modify boot qp to RTR
[../../src/mpid/ch3/channels/nemesis/netmod/ofa/ofa_init.c:2099] error(22): Could not modify boot qp to RTR
An error has occurred in cfx5solve:

The ANSYS CFX solver exited with return code 1.   No results file has been
created.

IBM MPI Distributed Parallel
doesn't work:

 cfx5remote: Rank 0:2: MPI_Init_thread: ibv_modify_qp(2rtr) failed
cfx5remote: Rank 0:2: MPI_Init_thread: ibv_modify_qpstate() failed
cfx5remote: Rank 0:2: MPI_Init_thread: Internal Error: Processes cannot connect to rdma device
cfx5remote: Rank 0:3: MPI_Init_thread: ibv_modify_qp(2rtr) failed
cfx5remote: Rank 0:3: MPI_Init_thread: ibv_modify_qpstate() failed
cfx5remote: Rank 0:3: MPI_Init_thread: Internal Error: Processes cannot connect to rdma device
cfx5remote: Rank 0:0: MPI_Init_thread: ibv_modify_qp(2rtr) failed
cfx5remote: Rank 0:0: MPI_Init_thread: ibv_modify_qpstate() failed
cfx5remote: Rank 0:0: MPI_Init_thread: Internal Error: Processes cannot connect to rdma device
An error has occurred in cfx5remote on itsm-clust827:

/ansys_inc/v191/CFX/bin/linux-amd64/ifort/solver-mpi.exe exited with return
code 1.

An error has occurred in cfx5remote on itsm-clust827:

/ansys_inc/v191/CFX/bin/linux-amd64/ifort/solver-mpi.exe exited with return
code 1.

MPI Application rank 0 exited before MPI_Finalize() with status 2
An error has occurred in cfx5remote on itsm-clust819:

/ansys_inc/v191/CFX/bin/linux-amd64/ifort/solver-mpi.exe exited with return
code 1.

An error has occurred in cfx5remote on itsm-clust819:

/ansys_inc/v191/CFX/bin/linux-amd64/ifort/solver-mpi.exe exited with return
code 1.

An error has occurred in cfx5solve:

The ANSYS CFX solver exited with return code 2.   No results file has been
created.

Going back to 18.2 solves the problem but only for Intel MPI Distributed Parallel.

Is anybody else experiencing this problem and knows of a workaround? I know CentOS is only officially supported up to version 7.4, but until now it worked flawlessly in 7.6 and 7.4 is EOL since last year already.

Does anybody know what changed in this kernel update that could break MPI?

Regards

Michael