CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Installation (http://www.cfd-online.com/Forums/openfoam-installation/)
-   -   System OpenMPI on RHEL Cluster (http://www.cfd-online.com/Forums/openfoam-installation/93341-system-openmpi-rhel-cluster.html)

pizzaice October 12, 2011 08:07

System OpenMPI on RHEL Cluster
 
Hi all,

I am trying to get OF 2.0.1 to run on our RHEL cluster. However, I am facing issues when trying to compile OF with the SYSTEMOPENMPI setting.

I have changed to SYSTEMOPENMPI in the bashrc. However, when I source it I get:
Code:

/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

I then adjusted preferences.sh with adding
export MPI_HOME=/opt/platform_mpi/
export MPI_ARCH_PATH=/opt/platform_mpi/

But I am getting the following error when compiling:
Code:

========================================
Done ThirdParty Allwmake
========================================

+ wmakePrintBuild -check
no git description found
+ /bin/rm -f OpenFOAM/Make/linux64Gcc44DPOpt/global.C OpenFOAM/Make/linux64Gcc44DPOpt/global.o
+ wmakeLnInclude OpenFOAM
+ wmakeLnInclude OSspecific/POSIX
+ Pstream/Allwmake
+ wmake libso dummy
'/opt/OpenFOAM/OpenFOAM-2.0.1/platforms/linux64Gcc44DPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

Can anyone help please?

Thanks

wyldckat October 12, 2011 17:34

Hi pizzaice,

I don't know any better way to say this, but that doesn't look good... not good at all. According to your description and the error messages, it would seem that you have incompatible binaries running in your environment. My first guess would be the MPI binaries, namely mpirun that isn't built properly for your system.

What happens when you run mpirun in a terminal? Does it give out the same error you've gotten?

Another big problem is that Gcc 4.1.2 does not work with OpenFOAM! At least not without some serious tinkering...
You'll have to request your system administrator to install another Gcc version more recent than 4.3.2.

Another possibility, since you are using RHEL, is the CentFOAM project: http://sourceforge.net/apps/mediawik...itle=Main_Page

Best regards and good luck!
Bruno

pizzaice November 16, 2011 12:57

Hi Bruno,

Thanks very much for the advice. Yes, you were right, it was an issue with openmpi.. I have now used gcc45 to build the sources and a newly compiled openmpi .. This seems to have fixed the issue!

Thanks again and happy foaming :-)

Chris

kjmaki October 26, 2012 11:25

compile 2.1.x with SYSTEMOPENMPI
 
I have a similar problem compiling openfoam 2.1.x on a cluster using the SYSTEMOPENMPI. Everything compiles except for the Pstream. Here is the output from the Allwmake in the src/Pstream directory:

login2$ ./Allwmake
+ wmake libso dummy
'/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
SOURCE=UOPwrite.C ; g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -IlnInclude -I. -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o
UOPwrite.C:29:17: error: mpi.h: No such file or directory
In file included from UOPwrite.C:32:
PstreamGlobals.H:55: error: ‘MPI_Request’ was not declared in this scope
PstreamGlobals.H:55: error: template argument 1 is invalid
PstreamGlobals.H:55: error: invalid type in declaration before ‘;’ token
UOPwrite.C: In static member function ‘static bool Foam::UOPstream::write(Foam::UPstream::commsTypes, int, const char*, std::streamsize, int)’:
UOPwrite.C:61: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:64: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:65: error: ‘MPI_Bsend’ was not declared in this scope
UOPwrite.C:81: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:84: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:85: error: ‘MPI_Send’ was not declared in this scope
UOPwrite.C:97: error: ‘MPI_Request’ was not declared in this scope
UOPwrite.C:97: error: expected ‘;’ before ‘request’
UOPwrite.C:103: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:106: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:107: error: ‘request’ was not declared in this scope
UOPwrite.C:108: error: ‘MPI_Isend’ was not declared in this scope
UOPwrite.C:115: error: request for member ‘size’ in ‘Foam::PstreamGlobals::outstandingRequests_’, which is of non-class type ‘int’
UOPwrite.C:119: error: request for member ‘append’ in ‘Foam::PstreamGlobals::outstandingRequests_’, which is of non-class type ‘int’
make: *** [Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o] Error 1

I have tried to customize the settings.sh file to properly set the PLIBS and PINC variables. My system-version of mpicc does not have --showme:compile and --showme:link options, so I have used the mpicc -compile-info to generate the paths to include and library directories and compiler flags. This is how my settings.sh file appears:


export FOAM_MPI=openmpi-system

# Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
#kjm export PINC="`mpicc --showme:compile`"
#kjm export PLIBS="`mpicc --showme:link`"
export PINC=-I/opt/apps/gcc4_4/mvapich2/1.6/include
export PLIBS='-L/opt/apps/gcc4_4/mvapich2/1.6/lib
-L/opt/apps/limic2/0.5.4//lib
-L/opt/ofed/lib64/
-lmpich -lopa -llimic2 -lpthread
-lrdmacm -libverbs -libumad -ldl -lrt'

#kjm libDir=`echo "$PLIBS" | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir=/opt/apps/gcc4_4/mvapich2/1.6/lib

# Bit of a hack: strip off 'lib' and hope this is the path to openmpi
# include files and libraries.
export MPI_ARCH_PATH="${libDir%/*}"
export MPI_HOME=MPI_ARCH_PATH

Any help is greatly appreciated!

Thanks!

Kevin

wyldckat October 26, 2012 19:43

Greetings Kevin,

From the output you've provided, it looks like you didn't update the shell environment for it to reflect the changes you've made to "settings.sh". I say this because the following line of output does not show your custom settings:
Quote:

Code:

SOURCE=UOPwrite.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra  -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX  -IlnInclude -I.  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude    -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o

The simplest way is to start a new terminal and try again. You can confirm if the settings you've made are active by running:
Code:

echo $PINC
echo $PLIBS

Best regards,
Bruno

kjmaki October 27, 2012 09:29

Thanks for your interest Bruno,

I have checked that the values of the shell env variables are as I set them in the settings.sh file. It still doesnt work. When I look in the Pstream/mpi/Make/options file, I see that I need a PFLAGS variable. When I echo $PFLAGS, it is empty. I grep for PFLAGS in the etc directory, and I don't see where it is set. Do you think this is what is causing my problem?

wyldckat October 27, 2012 12:57

Hi Kevin,

"PFLAGS" is loaded via these two lines on the "Make/options" file:
Code:

sinclude $(GENERAL_RULES)/mplib$(WM_MPLIB)
sinclude $(RULES)/mplib$(WM_MPLIB)

This means that this flag comes from "wmake/rules/General/mplibSYSTEMOPENMPI", which has got this:
Code:

# use the system installed openmpi - mpicc must be installed

PFLAGS    = -DOMPI_SKIP_MPICXX
PINC      = $(shell mpicc --showme:compile)
PLIBS      = $(shell mpicc --showme:link)

:eek: Well that explains the problems you're having!! Try commenting out or removing the "PINC" and "PLIBS" lines and you should be good to go!

Best regards,
Bruno

kjmaki October 28, 2012 10:13

Thanks for the hand Bruno,

Your suggestion was necessary for me to get it to work. I also had the issue that the -showme:compile option is not available for the mpicc on this cluster. The -compile-info option is a close alternative, although the output with this flag includes extra info. Also, the 'mpicc -compile-info' and 'mpicc -link-info' commands produce the exact same output. Is this normal?

I was able to test a parallel case, and it ran, although it has error messages on exit that I suspect may be due to me incorrectly specifying the PINC and PLIBS variables.

End

Finalising parallel run
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.

================================================== ===================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================== ===================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)

wyldckat October 28, 2012 14:57

Hi Kevin,

I suggest that you check the other files for ideas on how to configure "PINC" and "PLIBS":
Code:

ls -l $WM_PROJECT_DIR/wmake/rules/General/mplib*
By the way, I reported about the duplicate entries of the two variables in "settings.sh" and in the "mplibSYSTEMOPENMPI" files: http://www.openfoam.org/mantisbt/view.php?id=673 - the official fix is to remove these two variables from "settings.sh" and define them only in the "mplibSYSTEMOPENMPI" file.
Said fix is this one: https://github.com/OpenFOAM/OpenFOAM...10f50d61d5793e

Best regards,
Bruno


All times are GMT -4. The time now is 02:43.