CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

System OpenMPI on RHEL Cluster

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By pizzaice

Reply
 
LinkBack Thread Tools Display Modes
Old   October 12, 2011, 07:07
Default System OpenMPI on RHEL Cluster
  #1
New Member
 
Join Date: Nov 2009
Posts: 15
Rep Power: 7
pizzaice is on a distinguished road
Hi all,

I am trying to get OF 2.0.1 to run on our RHEL cluster. However, I am facing issues when trying to compile OF with the SYSTEMOPENMPI setting.

I have changed to SYSTEMOPENMPI in the bashrc. However, when I source it I get:
Code:
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
I then adjusted preferences.sh with adding
export MPI_HOME=/opt/platform_mpi/
export MPI_ARCH_PATH=/opt/platform_mpi/

But I am getting the following error when compiling:
Code:
========================================
Done ThirdParty Allwmake
========================================

+ wmakePrintBuild -check
no git description found
+ /bin/rm -f OpenFOAM/Make/linux64Gcc44DPOpt/global.C OpenFOAM/Make/linux64Gcc44DPOpt/global.o
+ wmakeLnInclude OpenFOAM
+ wmakeLnInclude OSspecific/POSIX
+ Pstream/Allwmake
+ wmake libso dummy
'/opt/OpenFOAM/OpenFOAM-2.0.1/platforms/linux64Gcc44DPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
Can anyone help please?

Thanks
y_jiang likes this.
pizzaice is offline   Reply With Quote

Old   October 12, 2011, 16:34
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,258
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi pizzaice,

I don't know any better way to say this, but that doesn't look good... not good at all. According to your description and the error messages, it would seem that you have incompatible binaries running in your environment. My first guess would be the MPI binaries, namely mpirun that isn't built properly for your system.

What happens when you run mpirun in a terminal? Does it give out the same error you've gotten?

Another big problem is that Gcc 4.1.2 does not work with OpenFOAM! At least not without some serious tinkering...
You'll have to request your system administrator to install another Gcc version more recent than 4.3.2.

Another possibility, since you are using RHEL, is the CentFOAM project: http://sourceforge.net/apps/mediawik...itle=Main_Page

Best regards and good luck!
Bruno
wyldckat is offline   Reply With Quote

Old   November 16, 2011, 12:57
Default
  #3
New Member
 
Join Date: Nov 2009
Posts: 15
Rep Power: 7
pizzaice is on a distinguished road
Hi Bruno,

Thanks very much for the advice. Yes, you were right, it was an issue with openmpi.. I have now used gcc45 to build the sources and a newly compiled openmpi .. This seems to have fixed the issue!

Thanks again and happy foaming :-)

Chris
pizzaice is offline   Reply With Quote

Old   October 26, 2012, 10:25
Default compile 2.1.x with SYSTEMOPENMPI
  #4
Member
 
Kevin Maki
Join Date: Mar 2009
Location: Ann Arbor, MI, USA
Posts: 41
Rep Power: 8
kjmaki is on a distinguished road
I have a similar problem compiling openfoam 2.1.x on a cluster using the SYSTEMOPENMPI. Everything compiles except for the Pstream. Here is the output from the Allwmake in the src/Pstream directory:

login2$ ./Allwmake
+ wmake libso dummy
'/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/dummy/libPstream.so' is up to date.
+ case "$WM_MPLIB" in
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers

wmake libso mpi
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
SOURCE=UOPwrite.C ; g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -IlnInclude -I. -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o
UOPwrite.C:29:17: error: mpi.h: No such file or directory
In file included from UOPwrite.C:32:
PstreamGlobals.H:55: error: ‘MPI_Request’ was not declared in this scope
PstreamGlobals.H:55: error: template argument 1 is invalid
PstreamGlobals.H:55: error: invalid type in declaration before ‘;’ token
UOPwrite.C: In static member function ‘static bool Foam::UOPstream::write(Foam::UPstream::commsTypes, int, const char*, std::streamsize, int)’:
UOPwrite.C:61: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:64: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:65: error: ‘MPI_Bsend’ was not declared in this scope
UOPwrite.C:81: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:84: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:85: error: ‘MPI_Send’ was not declared in this scope
UOPwrite.C:97: error: ‘MPI_Request’ was not declared in this scope
UOPwrite.C:97: error: expected ‘;’ before ‘request’
UOPwrite.C:103: error: ‘MPI_PACKED’ was not declared in this scope
UOPwrite.C:106: error: ‘MPI_COMM_WORLD’ was not declared in this scope
UOPwrite.C:107: error: ‘request’ was not declared in this scope
UOPwrite.C:108: error: ‘MPI_Isend’ was not declared in this scope
UOPwrite.C:115: error: request for member ‘size’ in ‘Foam::PstreamGlobals:utstandingRequests_’, which is of non-class type ‘int’
UOPwrite.C:119: error: request for member ‘append’ in ‘Foam::PstreamGlobals:utstandingRequests_’, which is of non-class type ‘int’
make: *** [Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o] Error 1

I have tried to customize the settings.sh file to properly set the PLIBS and PINC variables. My system-version of mpicc does not have --showme:compile and --showme:link options, so I have used the mpicc -compile-info to generate the paths to include and library directories and compiler flags. This is how my settings.sh file appears:


export FOAM_MPI=openmpi-system

# Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
#kjm export PINC="`mpicc --showme:compile`"
#kjm export PLIBS="`mpicc --showme:link`"
export PINC=-I/opt/apps/gcc4_4/mvapich2/1.6/include
export PLIBS='-L/opt/apps/gcc4_4/mvapich2/1.6/lib
-L/opt/apps/limic2/0.5.4//lib
-L/opt/ofed/lib64/
-lmpich -lopa -llimic2 -lpthread
-lrdmacm -libverbs -libumad -ldl -lrt'

#kjm libDir=`echo "$PLIBS" | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir=/opt/apps/gcc4_4/mvapich2/1.6/lib

# Bit of a hack: strip off 'lib' and hope this is the path to openmpi
# include files and libraries.
export MPI_ARCH_PATH="${libDir%/*}"
export MPI_HOME=MPI_ARCH_PATH

Any help is greatly appreciated!

Thanks!

Kevin
kjmaki is offline   Reply With Quote

Old   October 26, 2012, 18:43
Default
  #5
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,258
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Kevin,

From the output you've provided, it looks like you didn't update the shell environment for it to reflect the changes you've made to "settings.sh". I say this because the following line of output does not show your custom settings:
Quote:
Code:
SOURCE=UOPwrite.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra  -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3   -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX  -IlnInclude -I.  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude  -I/work/02220/kjmaki/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude    -fPIC -c $SOURCE -o Make/linux64GccDPOptSYSTEMOPENMPI/UOPwrite.o
The simplest way is to start a new terminal and try again. You can confirm if the settings you've made are active by running:
Code:
echo $PINC
echo $PLIBS
Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 27, 2012, 08:29
Default
  #6
Member
 
Kevin Maki
Join Date: Mar 2009
Location: Ann Arbor, MI, USA
Posts: 41
Rep Power: 8
kjmaki is on a distinguished road
Thanks for your interest Bruno,

I have checked that the values of the shell env variables are as I set them in the settings.sh file. It still doesnt work. When I look in the Pstream/mpi/Make/options file, I see that I need a PFLAGS variable. When I echo $PFLAGS, it is empty. I grep for PFLAGS in the etc directory, and I don't see where it is set. Do you think this is what is causing my problem?
kjmaki is offline   Reply With Quote

Old   October 27, 2012, 11:57
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,258
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Kevin,

"PFLAGS" is loaded via these two lines on the "Make/options" file:
Code:
sinclude $(GENERAL_RULES)/mplib$(WM_MPLIB)
sinclude $(RULES)/mplib$(WM_MPLIB)
This means that this flag comes from "wmake/rules/General/mplibSYSTEMOPENMPI", which has got this:
Code:
# use the system installed openmpi - mpicc must be installed

PFLAGS     = -DOMPI_SKIP_MPICXX
PINC       = $(shell mpicc --showme:compile)
PLIBS      = $(shell mpicc --showme:link)
Well that explains the problems you're having!! Try commenting out or removing the "PINC" and "PLIBS" lines and you should be good to go!

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 28, 2012, 09:13
Default
  #8
Member
 
Kevin Maki
Join Date: Mar 2009
Location: Ann Arbor, MI, USA
Posts: 41
Rep Power: 8
kjmaki is on a distinguished road
Thanks for the hand Bruno,

Your suggestion was necessary for me to get it to work. I also had the issue that the -showme:compile option is not available for the mpicc on this cluster. The -compile-info option is a close alternative, although the output with this flag includes extra info. Also, the 'mpicc -compile-info' and 'mpicc -link-info' commands produce the exact same output. Is this normal?

I was able to test a parallel case, and it ran, although it has error messages on exit that I suspect may be due to me incorrectly specifying the PINC and PLIBS variables.

End

Finalising parallel run
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.
interFoam: /home1/0000/build/admin/rpms/lonestar42/BUILD/mvapich2-1.6/src/mpid/ch3/channels/common/src/memory/mem_hooks.c:160: mvapich2_mfin: Assertion `1 == mem_hook_init' failed.

================================================== ===================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================== ===================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
kjmaki is offline   Reply With Quote

Old   October 28, 2012, 13:57
Default
  #9
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,258
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Kevin,

I suggest that you check the other files for ideas on how to configure "PINC" and "PLIBS":
Code:
ls -l $WM_PROJECT_DIR/wmake/rules/General/mplib*
By the way, I reported about the duplicate entries of the two variables in "settings.sh" and in the "mplibSYSTEMOPENMPI" files: http://www.openfoam.org/mantisbt/view.php?id=673 - the official fix is to remove these two variables from "settings.sh" and define them only in the "mplibSYSTEMOPENMPI" file.
Said fix is this one: https://github.com/OpenFOAM/OpenFOAM...10f50d61d5793e

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
CFX11 + Fortran compiler ? Mohan CFX 20 March 30, 2011 18:56
What is the best cluster file system to go with OF sampaio OpenFOAM Installation 4 March 17, 2011 05:10
How to install OpenFOAM15dev on RHEL AS3 cluster waynezw0618 OpenFOAM Installation 0 February 26, 2009 11:10
How to install OpenFOAM15dev on RHEL AS3 cluster waynezw0618 OpenFOAM Installation 0 February 26, 2009 11:04
Installation on a cluster running a queueing system PBS sampaio OpenFOAM Installation 12 June 20, 2005 17:16


All times are GMT -4. The time now is 05:00.