[syavash]

Hi All,

I am trying to run a custom solver in parallel on OpenFOAM 5. OF is installed on a cluster (root) and I have successfully managed to run one of its tutorial in parallel. However, when I try to run the custom solver in parallel I get the following error:

mpprun info: Starting impi run on 1 node ( 4 rank X 1 th ) for job ID 3202240



--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function static bool Foam::UPstream::init(int &, char **&)
    in file UPstream.C at line 37.

FOAM exiting

I have already added some flags to the Make/option to link the solver to the mpi library, according to the following thread:

wmake - options for parallel code using omp.h

ptFLAGS += -Nmpi -Nmkl -xCORE-AVX512

Could any body help me to resolve this issue?

Regards,
Syavash

[jherb]

What happens, if you start a standard OpenFOAM solver in parallel, e.g. simpleFoam in the motorbike tutorial?

[syavash]

Originally Posted by jherb

What happens, if you start a standard OpenFOAM solver in parallel, e.g. simpleFoam in the motorbike tutorial?

Hi Joachim,

As I stated in the original post, it goes well when running a tutorial in parallel. The problem is when I build a modified solver based on a native solver, it can't be run in parallel.

[fertinaz]

Hello Ehsan

I could not find any information in the link regarding the options you added to your makefiles. That guy in the link is probably using openmp related stuff in his code, therefore he wants to add its flag. I don't think you need to add AVX or MKL to ptFlags unless your modified solver taking advantage of vectorization or uses MKL.

By the way imposing vectorization on OF might even slow down your computations according to https://www.ixpug.org/images/docs/IX...g-OpenFOAM.pdf. See pages 27 to 32.

Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.

// Fatih

[syavash]

Dear Fatih,

Thanks for your reply.

Originally Posted by fertinaz

Hello Ehsan

I could not find any information in the link regarding the options you added to your makefiles. That guy in the link is probably using openmp related stuff in his code, therefore he wants to add its flag. I don't think you need to add AVX or MKL to ptFlags unless your modified solver taking advantage of vectorization or uses MKL.

Yes, the flags I added to my case were not included in that link. The -Nmpi flags was, however, necessary when I compiled non-root OpenFOAM and wanted to link it to the mpi library of the Cluster. Also, I thought that ptFlags is the proper way of adding the flags to the Make/options. At least, it did not complain when I compiled the code!

Please see the following link:
Non-root installation of OpenFOAM 2.4.x, parallel issue

By the way imposing vectorization on OF might even slow down your computations according to https://www.ixpug.org/images/docs/IX...g-OpenFOAM.pdf. See pages 27 to 32.

Thanks for the info. I added MKL and AVX according to the recommendation of the HPC administration, based on the HPC architecture. I have not yet examined the impact of AVX on the speed.

Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.

I will certainly try that. I will post the results here, and kindly ask for your opinion on the solution.

Kind Regards,
Syavash

[syavash]

Originally Posted by fertinaz

Anyway to solve the issue, I'd suggest you checking dependencies of your solver using ldd tool -- ldd `which yourSolverName`. It sounds like binaries from FOAM_USER_APPBIN are somehow linked against the dummy mpi which is located under FOAM_LIB. If that's the case, ldd will tell you, updating LD_LIBRARY_PATH should resolve your problem.

// Fatih

Dear Fatih,

I tried to use ldd as you suggested:

ldd 'which myInterFoam'

However, I get the following error:

ldd: ./which myInterFoam: No such file or directory

Is this the correct way of using ldd?

Kind Regards,
Syavash

[fertinaz]

Nope, you need to use backticks and you don't need ./ since which is already in your PATH:

ldd `which myIcoFoam`

[syavash]

Originally Posted by fertinaz

Nope, you need to use backticks and you don't need ./ since which is already in your PATH:

ldd `which myIcoFoam`

Thanks for the correction. I executed the command,

ldd `which myInterFoam`

I get the following lines which I cannot figure out!

        linux-vdso.so.1 =>  (0x00007ffee71a3000)
        libimmiscibleIncompressibleTwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libimmiscibleIncompressibleTwoPhaseMixture.so (0x00007fd5f631f000)
        libturbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libturbulenceModels.so (0x00007fd5f5d97000)
        libincompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTurbulenceModels.so (0x00007fd5f571f000)
        libfiniteVolume.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfiniteVolume.so (0x00007fd5f2e55000)
        libfvOptions.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfvOptions.so (0x00007fd5f29f2000)
        libmeshTools.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libmeshTools.so (0x00007fd5f2238000)
        libsampling.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsampling.so (0x00007fd5f1a69000)
        libOpenFOAM.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libOpenFOAM.so (0x00007fd5f0c81000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fd5f0a7d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fd5f077b000)
        libmpicxx.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpicxx.so.12 (0x00007fd5f055b000)
        libmpifort.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpifort.so.12 (0x00007fd5f01b2000)
        libmpi.so.12 => /.../.../.../.../.../impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpi.so.12 (0x00007fd5ef52d000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fd5ef325000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd5ef109000)
        libstdc++.so.6 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libstdc++.so.6 (0x00007fd5f659a000)
        libgcc_s.so.1 => /.../.../.../.../.../GCCcore/6.4.0/lib64/libgcc_s.so.1 (0x00007fd5f6582000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd5eed3c000)
        libincompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libincompressibleTransportModels.so (0x00007fd5eea61000)
        libinterfaceProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libinterfaceProperties.so (0x00007fd5ee7b2000)
        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)
        libtwoPhaseMixture.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseMixture.so (0x00007fd5ee383000)
        libtwoPhaseProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtwoPhaseProperties.so (0x00007fd5ee109000)
        libimf.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libimf.so (0x00007fd5edb7b000)
        libsvml.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libsvml.so (0x00007fd5ec4c8000)
        libirng.so => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libirng.so (0x00007fd5ec154000)
        libintlc.so.5 => /.../.../.../.../.../ifort/2018.1.163-GCC-6.4.0-2.28/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libintlc.so.5 (0x00007fd5ebee7000)
        libtriSurface.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libtriSurface.so (0x00007fd5ebc1a000)
        libcompressibleTurbulenceModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTurbulenceModels.so (0x00007fd5eb585000)
        libfluidThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfluidThermophysicalModels.so (0x00007fd5ead77000)
        libsurfMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsurfMesh.so (0x00007fd5ea9ec000)
        libfileFormats.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libfileFormats.so (0x00007fd5ea74a000)
        liblagrangian.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/liblagrangian.so (0x00007fd5ea519000)
        libdynamicMesh.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libdynamicMesh.so (0x00007fd5e9c37000)
        libconversion.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libconversion.so (0x00007fd5e998b000)
        libz.so.1 => /.../.../.../.../.../zlib/1.2.11-GCCcore-6.4.0/lib/libz.so.1 (0x00007fd5e9973000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd5f652e000)
        libcompressibleTransportModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libcompressibleTransportModels.so (0x00007fd5e976d000)
        libsolidThermo.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidThermo.so (0x00007fd5e9409000)
        libsolidSpecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libsolidSpecie.so (0x00007fd5e91c1000)
        libspecie.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libspecie.so (0x00007fd5e8afc000)
        libthermophysicalProperties.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libthermophysicalProperties.so (0x00007fd5e8821000)
        libextrudeModel.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libextrudeModel.so (0x00007fd5e85fd000)
        libreactionThermophysicalModels.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/libreactionThermophysicalModels.so (0x00007fd5e779c000)

Could be something wrong here??

Kind Regards,
Syavash

[fertinaz]

Yes something is wrong there. When you look carefully you'll see that your solver is linked against the dummy libPstream - the library that implements MPI comm in OF. See:

        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)

There must be another libPstream.so under an mpi directory in the same FOAM_LIB. Try to prepend that path to your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:LD_LIBRARY_PATH

${FOAM_LIBBIN}/mpi could be wrong, to make sure go to your OF installation path and run sth. like "find . -name libPstream.so" and check results.

Hope this helps

// Fatih

[syavash]

Originally Posted by fertinaz

Yes something is wrong there. When you look carefully you'll see that your solver is linked against the dummy libPstream - the library that implements MPI comm in OF. See:

        libPstream.so => /.../.../.../.../.../OpenFOAM/.../OpenFOAM-5.0-20180108/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00007fd5ee5ab000)

There must be another libPstream.so under an mpi directory in the same FOAM_LIB. Try to prepend that path to your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:LD_LIBRARY_PATH

${FOAM_LIBBIN}/mpi could be wrong, to make sure go to your OF installation path and run sth. like "find . -name libPstream.so" and check results.

Hope this helps

// Fatih

Thanks for the tip. I used the find command and I got the following results:

./platforms/linux64IccDPInt32Opt/lib/mpi/libPstream.so
./platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so

Could you tell me according to the results above what I should export as LD_LIBRARY_PATH now?

Kind Regards,
Syavash

[fertinaz]

So according to those results ${FOAM_LIBBIN}/mpi is correct. If you export dummy you'll have the same error again. Just type the command with export I've written in my previous reply before you call your solver or if you're using a job script add it to that script after you load OF modules. Do not add it to your bashrc.


Hope this helps

[syavash]

Originally Posted by fertinaz

So according to those results ${FOAM_LIBBIN}/mpi is correct. If you export dummy you'll have the same error again. Just type the command with export I've written in my previous reply before you call your solver or if you're using a job script add it to that script after you load OF modules. Do not add it to your bashrc.


Hope this helps

I really appreciate your help. I could get it to work! Thank you.

Only one point. I had to recompile my solver for this correction to work. Hope this help others with the same issue.

Kind Regards,
Syavash

P. S. It is a basic question, but what is LD_LIBRARY_PATH doing?

[fertinaz]

Ehsan

That's a basic question but it is in fact quite complicated to explain

I can recommend two references:
-- http://tldp.org/HOWTO/Program-Librar...libraries.html
-- "An Introduction to GCC" Book by Brian Gough

In short; when you use a shared library while writing your code, you need to tell the compiler that those libraries should be dynamically linked during the compilation. After you execute your code, parts of those libraries used in your code will be invoked during runtime.

This mechanism has a few advantages;
1- You don't need to integrate your libraries into the executables, so it keeps binaries in a bare minimum.
2- When those libs are updated, you don't need to compile everything from scratch - for instance if you change a boundary condition code in OF stack, you don't compile solvers from scratch because BCs are shared objects.
3- If a single shared lib is used by multiple executables, OS still keeps a single copy of that library in the memory which means reduced memory consumption.

How does this relate to LD_LIBRARY_PATH? It is the environment variable that keeps a list of those libraries checked by Linux OS runtime.

By the way, primary reason I'm writing this message is that I've seen a typo in the export command. Correct version should be:

export LD_LIBRARY_PATH=${FOAM_LIBBIN}/mpi:$LD_LIBRARY_PATH

Hope this is useful and I'd highly suggest reading the references above for further details.

// Fatih