CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Parallel processing problem with mpich

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   October 16, 2007, 19:25
Default Hi guys: I have been stuck
  #1
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Hi guys:

I have been stuck with a parallel processing on openfoam 1.4.1. I tested a case with 2.3 million cells. After tested several different node numbers, I found that a lot of time has been wasted by using openmpi. Then I decided to switch to other mpich packages on my school cluster. Here is what I did:

=================================================
export MPICH_PATH=/usr/global/mpich-infinicon
export MPICH_ARCH_PATH=$MPICH_PATH/platforms/$WM_OPTIONS
export MPICH_ROOT=$MPICH_ARCH_PATH

AddLib $MPICH_ARCH_PATH/lib
AddPath $MPICH_ARCH_PATH/bin

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich-infinicon
=================================================

The problem is I don't have permission to write in /global/mpich-infinicon. I am wondering if I really have to put "platforms/$WM_OPTIONS", "lib" and "bin" in the global directory or there is other way around. Thank you for help.

Ning
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 03:24
Default It is the AddLib & AddPath I a
  #2
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
It is the AddLib & AddPath I assume that give the problems since they do an 'mkdir' under the hood. Remove the mkdir from the AddLib and AddPath and see if that works.
mattijs is offline   Reply With Quote

Old   October 17, 2007, 10:25
Default I tried ./Allwmake in ..src/Ps
  #3
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
I tried ./Allwmake in ..src/Pstream directory. I got an error:
=================================================
+ '[' MPICH = LAM -o MPICH = OPENMPI -o MPICH = MPI -o MPICH = MPICH -o MPICH = MPICH-GM -o MPICH = MPIGAMMA ']'
+ export WM_OPTIONS=linux64GccDPOptMPICH
+ WM_OPTIONS=linux64GccDPOptMPICH
+ wmake libso mpi
/usr/bin/ld: cannot find -lmpich
collect2: ld returned 1 exit status
=================================================
Then I followed the link (http://openfoamwiki.net/non_cdl/Open...mpile_OpenFOAM)and installed binutil-2.18. I got the error as follows:
=================================================
+ '[' MPICH = LAM -o MPICH = OPENMPI -o MPICH = MPI -o MPICH = MPICH -o MPICH = MPICH-GM -o MPICH = MPIGAMMA ']'
+ export WM_OPTIONS=linux64GccDPOptMPICH
+ WM_OPTIONS=linux64GccDPOptMPICH
+ wmake libso mpi
/home1/nzy102/OpenFOAM/linux64/binutils-2.18/bin/ld: cannot find -lmpich
collect2: ld returned 1 exit status
make: *** [/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPs tream.so] Error 1
=================================================
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 11:27
Default Check whether /usr/global/mpic
  #4
Senior Member
 
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 698
Rep Power: 12
msrinath80 is on a distinguished road
Check whether /usr/global/mpich-infinicon/lib contains *.a (static) or *.so (shared) libraries. I am guessing that it contains only *.a files. To build libPstream.so successfully, you need *.so files in there. Mattijs, please correct me if I am wrong.
msrinath80 is offline   Reply With Quote

Old   October 17, 2007, 12:02
Default Thanks,Srinath. I changed my .
  #5
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Thanks,Srinath. I changed my .bashrc and now I made a new libPstream.so. This is what I used in my .bashrc:

=================================================

# export MPICH_VERSION=infinicon-3.1.0.0.35G
export MPICH_PATH=/usr/global/mpich-infinicon
export MPICH_ARCH_PATH=$MPICH_PATH
# export MPICH_ARCH_PATH=$MPICH_PATH/platforms/$WM_OPTIONS
export MPICH_ROOT=$MPICH_ARCH_PATH


# AddLib $MPICH_ARCH_PATH/lib
# AddPath $MPICH_ARCH_PATH/bin

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich-infinicon
# export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mvapich-$MPICH_VERSION

=================================================

But when I submit my jobs using pbs file, I got an error:

=================================================
icoFoam: symbol lookup error: /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined symbol: FStatusText
icoFoam: symbol lookup error: /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined symbol: FStatusText
=================================================

Anybody knows what this means? For the new mpich, what command I should use for parallel?
Should I still use something like:

mpirun -np 2 icoFoam /home1/nzy102/OpenFOAM/nzy102-1.4.1/run/tutorials/icoFoam lvad_90_0_5_grid2_np2 -parallel

Thank you.

Ning
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 12:12
Default Replace icoFoam with `which ic
  #6
Senior Member
 
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 698
Rep Power: 12
msrinath80 is on a distinguished road
Replace icoFoam with `which icoFoam` and it should work. MPICH usually needs the full path to icoFoam specified when using the mpirun command. e.g:

mpirun -np 2 `which icoFoam` /home1/nzy102/OpenFOAM/nzy102-1.4.1/run/tutorials/icoFoam lvad_90_0_5_grid2_np2 -parallel
msrinath80 is offline   Reply With Quote

Old   October 17, 2007, 12:15
Default You may also need to specify a
  #7
Senior Member
 
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 698
Rep Power: 12
msrinath80 is on a distinguished road
You may also need to specify a machinefile:

mpirun -np 2 -machinefile mymachinefile `which icoFoam_1` . mycase -parallel

Where mymachinefile contains:
hostname:number of processors

e.g.:
ning:2

Or you can give $PBS_NODEFILE instead of mymachinefile. e.g.:

mpirun -np 2 -machinefile $PBS_NODEFILE `which icoFoam_1` . mycase -parallel
msrinath80 is offline   Reply With Quote

Old   October 17, 2007, 12:25
Default Oh, and after you change to a
  #8
Senior Member
 
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 698
Rep Power: 12
msrinath80 is on a distinguished road
Oh, and after you change to a different MPI library, go to your application source and recompile the solver. For icoFoam, you need to go to ~/OpenFOAM/OpenFOAM-1.4/applications/solvers/incompressible/icoFoam/

and execute wclean && wmake there. That should take care of your symbol lookup errors, I think?
msrinath80 is offline   Reply With Quote

Old   October 17, 2007, 12:45
Default Thank you for quick replied, S
  #9
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Thank you for quick replied, Srinath. I compiled my icoFoam and got:
=================================================
Making dependency list for source file icoFoam.C
SOURCE=icoFoam.C ; g++ -m64 -Dlinux64 -DDP -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -march=opteron -O3 -DNoRepository -ftemplate-depth-40 -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/finiteVolume/lnInclude -IlnInclude -I. -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/OpenFOAM/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOpt/icoFoam.o
In file included from icoFoam.C:105:
computeMassFlux.H:17:2: warning: no newline at end of file
g++ -m64 -Dlinux64 -DDP -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -march=opteron -O3 -DNoRepository -ftemplate-depth-40 -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/finiteVolume/lnInclude -IlnInclude -I. -I/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/src/OpenFOAM/lnInclude -fPIC Make/linux64GccDPOpt/icoFoam.o -L/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt \
-lfiniteVolume -lOpenFOAM -liberty -ldl -lm -o /home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux64GccDPOpt/icoFoam
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `FStatusText'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_prepare'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_wc_status_msg'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_clear_async_event_handler'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_get_iba_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_create_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `FCmStatusText'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_event_syndrome_sym'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_clear_comp_eventh'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_deregister_mr'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_query_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_set_async_event_handler'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_alloc_pd'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_strerror'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_disconnect'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_poll_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_poll_cq_wait'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_finalize'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_create_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_destroy_cq'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_post_recv2'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_post_send2'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_destroy_qp'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_register_mr'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `EVAPI_set_comp_eventh'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `iba_wc_type_msg'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_event_record_sym'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_dealloc_pd'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_init'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `ib_vapi_cm_connect'
/home1/nzy102/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/mpich-infinicon/libPst ream.so: undefined reference to `VAPI_get_iba_qp'
=================================================
Something is messed up with my new libpstream.so.

Ning
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 12:53
Default computeMassFlux.H is not a sta
  #10
Senior Member
 
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 698
Rep Power: 12
msrinath80 is on a distinguished road
computeMassFlux.H is not a standard part of icoFoam is it? Are you sure that it has been parallelized properly?

Try this. Recompile the original icoFoam solver and see if you get these errors. I'm sorry I cannot offer more guidance. I don't know what could be wrong. Please wait for Mattijs to respond. He is the parallel expert
msrinath80 is offline   Reply With Quote

Old   October 17, 2007, 13:03
Default Srinath: ComputeMassFlux wo
  #11
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Srinath:

ComputeMassFlux works with openmpi. You can ignore the computemassflux. Thank you for all the suggestions.

Mattijs:

Can you take a look at this case when you get a chance?

Thank you.
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 15:59
Default Hello Ning, 1: By commentin
  #12
Senior Member
 
Martin Beaudoin
Join Date: Mar 2009
Posts: 330
Rep Power: 13
mbeaudoin will become famous soon enough
Hello Ning,

1: By commenting out
# AddLib $MPICH_ARCH_PATH/lib

you won't be able to run your MPI application because your LD_LIBRARY_PATH environment variable will not be set properly to find the MPI runtime libraries needed.

2: By commenting out
# AddPath $MPICH_ARCH_PATH/bin

you won't be able to use the proper version of mpirun associate with your local version of MPI. You might (and probably will) run into problems.

3: About your compilation problems and the numerous error messages associated to libPstream:

You are probably missing one or more libraries that need to be specified with libmpich.

Find someone at your location who has already compiled an application with your local version of MPICH, and look at the list of libraries that need to be specified for linking such an application.

Martin
mbeaudoin is offline   Reply With Quote

Old   October 17, 2007, 19:15
Default Hi Martin: I linked the lib
  #13
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Hi Martin:

I linked the library required to icofoam solver and now it works fine. I still have addlib and addpath commented out and hasn't seen any problem so far. Since I don't have permission to write in the global directory, I am gonna leave it there. Thank you.

Ning
nzy102 is offline   Reply With Quote

Old   October 17, 2007, 22:45
Default Hello Ning, For your inform
  #14
Senior Member
 
Martin Beaudoin
Join Date: Mar 2009
Posts: 330
Rep Power: 13
mbeaudoin will become famous soon enough
Hello Ning,

For your information, you don't need write access to your global /global/mpich-infinicon directory.

I am using a local openmpi version on my cluster, and I don't have write access there either. Don't need to.

Addlib() and AddPath() are defined in the file $WM_PROJECT_DIR/.bashrc.

Take a look at their definition:

AddPath(){
export PATH=$1:$PATH
if [ ! -d $1 ]; then
mkdir -p $1
fi
}

AddLib(){
export LD_LIBRARY_PATH=$1:$LD_LIBRARY_PATH
if [ ! -d $1 ]; then
mkdir -p $1
fi
}

See the lines in red? What this means is that if the path used as parameter to the functions AddLib and AddPath don't exists, then mkdir will try to create it.

So if you use a valid path when calling AddLib and AddPath, mkdir will never ever get called in the first place.

Check the validity of your paths when using AddLib and AddPath.

From your example, the paths you are using are probably wrong in the first place because
$MPICH_ARCH_PATH/lib would expand to
$MPICH_PATH/platforms/$WM_OPTIONS/lib which would probably looks like
/usr/global/mpich-infinicon/platforms/linuxGccDPOpt/lib.

I doubt this is a valid path on your system, hence the problem with mkdir when using AddLib.

What you probably want instead in your .bashrc is :

export MPICH_ROOT=$MPICH_PATH
AddLib $MPICH_PATH/lib
AddPath $MPICH_PATH/bin

Your cluster sysadmin probably set up your initial user environment properly for using the /global/mpich-infinicon libraries and binaries by default, this is why your setup does work even though the two important lines Addlib and AddPath are commented out.

Martin
mbeaudoin is offline   Reply With Quote

Old   October 18, 2007, 00:05
Default Now everything is clear to me.
  #15
Member
 
Ning Yang
Join Date: Mar 2009
Location: University Park, PA, USA
Posts: 85
Rep Power: 8
nzy102 is on a distinguished road
Now everything is clear to me. Thank you so much.

Ning
nzy102 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Parallel processing problem with Fluent Harish FLUENT 3 February 8, 2011 10:49
MPICH parallel problem (CFX-11 preview 5) CFDworker CFX 8 October 10, 2006 21:53
parallel processing problem with multiproc machine hajra_rabaeto!!!!!!! FLUENT 2 January 28, 2006 20:11
Batch+Parallel+MPICH startup? KEN CFX 3 May 11, 2005 09:54
CFX-5.7 MPICH Parallel Problem (Output of Results) James Date CFX 7 February 15, 2005 17:03


All times are GMT -4. The time now is 06:25.