CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

Pstream on Cray

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   September 23, 2010, 09:51
Default Pstream on Cray
  #1
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Dear all,

I am trying to compile OpenFOAM-1.6 on a Cray XT5 machine and so far managed to compile it with help of this document. I also tried a static build, after this thread, but that didn't work out.
Now solvers etc work, but not in parallel. That kind of defeats the purpose of installation on a cluster, doesn't it? The trouble is with Pstream, it does not compile properly. I went back to ThirdParty and found out that I overlooked that metis-parmetis did not compile. When I compile $WM_THIRD_PARTY_DIR/ParMetis-3.1 ld gives an error that it can't find lpmi while compiling METISLib (which should build metis-parmetis.so). Would anyone have an idea how to fix this? Many thanks in advance!
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   September 23, 2010, 10:44
Default No mpirun?
  #2
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Ok, an update:

I found out that libpmi.so lives in /opt/cray/pmi/default/lib64, so I changed $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT from


Code:
PFLAGS     = 
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L$(MPICH_PATH)/pmi/lib -L$(MPICH_PATH)/util/lib -lmpich -lpmi -lalpslli -lalpsutil -lrt
to
Code:
PFLAGS     =
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt
to reflect the right paths (I hope). Now ThirdParty and Pstream compiled without error!! Whoohoo!

The problem I have now is that there is no mpirun, only mpigammarun and mpirunDebug. Perhaps a noob question, but does anyone know if this is normal or not and how to use mpigammarun/mpirunDebug? I mean, debug is slower, right?
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   September 25, 2010, 09:59
Default
  #3
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Gijsbert,

I don't have any experience with Crays (but I do wish I had ), but... pmi? "Process Manager Interface"? According to this presentation and also indicated by your posted code snippets, you're looking for MPICH, not pmi

So, if I'm not mistaken:
  1. You should take out all references in your mplibMPT file to pmi.
  2. Find where your system's MPICH lives, probably by running:
    Code:
    echo $MPICH_PATH
  3. Edit the file $WM_PROJECT_DIR/etc/settings.sh. Search for "mpich" and update the definitions it has for it.
  4. Edit the file $WM_PROJECT_DIR/etc/bashrc and make sure you set properly the variable WM_MPLIB to MPICH.
  5. Source the bashrc file:
    Code:
    . $WM_PROJECT_DIR/etc/bashrc
    or start a new terminal. I personally always start a new terminal... seems to be the most efficient away to make sure that the changes a properly implemented.
  6. Rebuild the Third Party tools/libraries and OpenFOAM, just to make sure things are set into place (note: if you feel the need to backup before this step, you should!):
    Code:
    cd $WM_THIRD_PARTY_DIR
    wclean all
    cd $WM_PROJECT_DIR
    wclean all
    Allwmake > make.log 2>&1
    This way you keep a log of the build process. It'll make your life easier to assert what went wrong
    NOTE: if while running "wclean all" in the ThirdParty folder there are some errors about "gmake not found", it's not really important either so you can ignore those errors
Note: the link for that presentation at the top of my post also has some interesting optimization tweaks for MPICH in CrayXT!

Good luck! Best regards,
Bruno

Last edited by wyldckat; September 25, 2010 at 10:04. Reason: I had forgotten that it was also necessary to rebuild the ThirdParty libraries, just in case.
wyldckat is online now   Reply With Quote

Old   September 27, 2010, 03:28
Default Solved
  #4
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Hi Bruno,

Thank you for your reply! The presentation you mentioned looks very useful, I wil definitely have a more detailed look at it.

Quote:
you're looking for MPICH, not pmi
Actually, I was looking for pmi, not MPICH . When I compiled ParMetis in ThirdParty and Pstream, the compiler complained that ld could not find -lpmi. The reason is that I needed to set up a new compiler in wmake/rules, called crayxt (well, of course I could call it banana, but crayxt seemed more appropriate). According to a friend of mine at CSC and this document a new MPI needs to be defined in settings.sh, namely:

Code:
MPT)
    export MPICH_PATH=$MPICHBASEDIR
    export MPI_ARCH_PATH=$MPICH_DIR
    export MPICH_ROOT=$MPI_ARCH_PATH
    _foamAddLib $MPI_ARCH_PATH/lib
    _foamAddLib $MPICH_PATH/pmi/lib
    _foamAddLib $MPICH_PATH/util/lib
    export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpt
    ;;
Now, the trouble is that on the system I was installing OF the pmi and alpsutil libraries moved to a different place. Therefore, they did not live under $MPICH_PATH anymore. The solution was to adjust the paths above to reflect the ones on the local machine and also edit $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT to look like:

Code:
PFLAGS     =
PINC       = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt
With these adjustments OF ThirdParty and Pstream compiled without any error .

Then, the other issue I mentioned above was how to use mpirun. After successfully compiling Pstream, there was still no mpirun. It turned out that after decomposing with decomposePar I could just write a qsub script to submit the job, where the executing line looks like
Code:
aprun -n <nr_of_processors> <solver> -parallel > log.txt
As simple as that! Now it's crunching time .
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   February 27, 2011, 23:17
Default
  #5
New Member
 
Join Date: Apr 2010
Posts: 17
Rep Power: 7
namCFD is on a distinguished road
Hello Gijs,

I am trying to install 1.7.x on Cray XT4 but running into trouble: /usr/bin/lid cannot find lOpenFOAM. Is it possible to for you to list what you have done so I can follow to install?

I am using c-shell (csh) instead of bash and PrgEnv-gnu instead of PrgEnv-pgi. gcc is 4.4.1

I was able to install 1.7.1 but it is not working in parallel with aprun. In 1.7.1, I can use mpirun.

Thank you for your help.
__________________
Nam
namCFD is offline   Reply With Quote

Old   February 28, 2011, 01:40
Default
  #6
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   March 15, 2011, 19:11
Default
  #7
New Member
 
Join Date: Apr 2010
Posts: 17
Rep Power: 7
namCFD is on a distinguished road
Hello Gijs,

After modifying your write-up to fit my system, I got the following error while running a batch job using "aprun": aprun -n 8 icoFoam -parallel with cavity tutorial. The error message is:

icoFoam: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

Do you have any hints on how to solve this error? I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.

Thanks,

Nam
__________________
Nam
namCFD is offline   Reply With Quote

Old   March 16, 2011, 03:30
Default
  #8
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Hi Nam,

Quote:
I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.
Ok, so what did you do exactly? You know where libstdc++.so.6 lives and you added it to LD_LIBRARY_PATH?
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   March 16, 2011, 15:33
Default
  #9
New Member
 
Join Date: Apr 2010
Posts: 17
Rep Power: 7
namCFD is on a distinguished road
Hi Gijs,

Update: I contacted the system administrator and indeed libstdc++.so.6 does not exist in the compute node. Do you have any ideas on how to solve this issue?
----------------------

First of all, thank you for your time and help.

I could not get OpenFOAM-1.7.x compiled correctly on Cray XT4 but OpenFOAM-1.7.1 is ok after using foamInstallationTest command. I don't know why.

I know that the libstdc++.so.6 lives in /usr/lib64 in my system. So, I did this in my batch script: export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH.

Do you think /usr/lib64 might only exist in the log-in node but not on compute nodes?

Thanks,
__________________
Nam

Last edited by namCFD; March 16, 2011 at 18:35. Reason: Update new info
namCFD is offline   Reply With Quote

Old   March 23, 2011, 17:07
Default
  #10
New Member
 
Join Date: Apr 2010
Posts: 17
Rep Power: 7
namCFD is on a distinguished road
Hello Gijs,

Can you help me with this error message?

bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

From function Pstream::init(int& argc, char**& argv)
in file Pstream.C at line 73.

Thanks,
__________________
Nam
namCFD is offline   Reply With Quote

Old   March 24, 2011, 02:36
Default
  #11
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 382
Rep Power: 9
gwierink is on a distinguished road
Hi Nam,

Quote:
bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor
This looks like a run time error, is that correct? If so, did you decompose the case with decomposePar?
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   March 24, 2011, 12:34
Default
  #12
New Member
 
Join Date: Apr 2010
Posts: 17
Rep Power: 7
namCFD is on a distinguished road
Hi Gijs,

I did decompose the case before submitting the case to HPC with aprun. However, we compiled OpenFOAM using openMPI that is in the ThirdParty folder. I think this is the problem because the compute nodes do not have openMPI, only MPICH2. Do you think this is the cause? We are trying to recompile with MPICH2 but as you already know, it is not a simple process. We are now trying to install OpenFOAM on Cray XE6. Do you have any pointers?

Thanks,
__________________
Nam
namCFD is offline   Reply With Quote

Old   March 11, 2014, 10:20
Default
  #13
New Member
 
Silvia Matt
Join Date: Aug 2013
Location: New Orleans
Posts: 1
Rep Power: 0
SMatt is on a distinguished road
Quote:
Originally Posted by gwierink View Post
Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.
I know this is an old thread, but I am struggling with getting OpenFOAM 2.1.1 to work in parallel on a Cray XE6. The links in the above post are broken.

Can anybody point me to any other threads on how to make the install work, or should I just follow the steps described above by gwierink?

Thanks.
Silvia
SMatt is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Pstream library error in parallel mode begou OpenFOAM Installation 9 June 21, 2015 15:46
OpenFOAM static build on Cray XT5 asaijo OpenFOAM Installation 9 April 6, 2011 12:21
Prallel run error Pstream library tsjb00 OpenFOAM Running, Solving & CFD 1 April 12, 2008 08:43
How to run and save tranisent case on Cray XD1 Leon FLUENT 0 October 3, 2006 21:59
HPF and cray T3E Frank Muldoon Main CFD Forum 1 August 4, 1998 05:41


All times are GMT -4. The time now is 04:34.