CFD Online Discussion Forums

CFD Online Discussion Forums (
-   OpenFOAM Installation (
-   -   Pstream on Cray (

gwierink September 23, 2010 09:51

Pstream on Cray
Dear all,

I am trying to compile OpenFOAM-1.6 on a Cray XT5 machine and so far managed to compile it with help of this document. I also tried a static build, after this thread, but that didn't work out.
Now solvers etc work, but not in parallel. That kind of defeats the purpose of installation on a cluster, doesn't it? ;) The trouble is with Pstream, it does not compile properly. I went back to ThirdParty and found out that I overlooked that metis-parmetis did not compile. When I compile $WM_THIRD_PARTY_DIR/ParMetis-3.1 ld gives an error that it can't find lpmi while compiling METISLib (which should build Would anyone have an idea how to fix this? Many thanks in advance!

gwierink September 23, 2010 10:44

No mpirun?
Ok, an update:

I found out that lives in /opt/cray/pmi/default/lib64, so I changed $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT from


PINC      = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L$(MPICH_PATH)/pmi/lib -L$(MPICH_PATH)/util/lib -lmpich -lpmi -lalpslli -lalpsutil -lrt


PINC      = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt

to reflect the right paths (I hope). Now ThirdParty and Pstream compiled without error!! :D Whoohoo!

The problem I have now is that there is no mpirun, only mpigammarun and mpirunDebug. Perhaps a noob question, but does anyone know if this is normal or not and how to use mpigammarun/mpirunDebug? I mean, debug is slower, right?

wyldckat September 25, 2010 09:59

Greetings Gijsbert,

I don't have any experience with Crays (but I do wish I had :rolleyes:), but... pmi? "Process Manager Interface"? According to this presentation and also indicated by your posted code snippets, you're looking for MPICH, not pmi ;)

So, if I'm not mistaken:
  1. You should take out all references in your mplibMPT file to pmi.
  2. Find where your system's MPICH lives, probably by running:

    echo $MPICH_PATH
  3. Edit the file $WM_PROJECT_DIR/etc/ Search for "mpich" and update the definitions it has for it.
  4. Edit the file $WM_PROJECT_DIR/etc/bashrc and make sure you set properly the variable WM_MPLIB to MPICH.
  5. Source the bashrc file:

    . $WM_PROJECT_DIR/etc/bashrc
    or start a new terminal. I personally always start a new terminal... seems to be the most efficient away to make sure that the changes a properly implemented.
  6. Rebuild the Third Party tools/libraries and OpenFOAM, just to make sure things are set into place (note: if you feel the need to backup before this step, you should!):

    wclean all
    wclean all
    Allwmake > make.log 2>&1

    This way you keep a log of the build process. It'll make your life easier to assert what went wrong ;)
    NOTE: if while running "wclean all" in the ThirdParty folder there are some errors about "gmake not found", it's not really important either so you can ignore those errors ;)
Note: the link for that presentation at the top of my post also has some interesting optimization tweaks for MPICH in CrayXT!

Good luck! Best regards,

gwierink September 27, 2010 03:28

Hi Bruno,

Thank you for your reply! The presentation you mentioned looks very useful, I wil definitely have a more detailed look at it.


you're looking for MPICH, not pmi
Actually, I was looking for pmi, not MPICH :). When I compiled ParMetis in ThirdParty and Pstream, the compiler complained that ld could not find -lpmi. The reason is that I needed to set up a new compiler in wmake/rules, called crayxt (well, of course I could call it banana, but crayxt seemed more appropriate). According to a friend of mine at CSC and this document a new MPI needs to be defined in, namely:


    _foamAddLib $MPI_ARCH_PATH/lib
    _foamAddLib $MPICH_PATH/pmi/lib
    _foamAddLib $MPICH_PATH/util/lib

Now, the trouble is that on the system I was installing OF the pmi and alpsutil libraries moved to a different place. Therefore, they did not live under $MPICH_PATH anymore. The solution was to adjust the paths above to reflect the ones on the local machine and also edit $WM_PROJECT_DIR/wmake/rules/crayxt/mplibMPT to look like:


PINC      = -I$(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/stat/1.1.0/lib/alps -lmpich -lpmi -lalpslli -lalpsutil -lrt

With these adjustments OF ThirdParty and Pstream compiled without any error :D.

Then, the other issue I mentioned above was how to use mpirun. After successfully compiling Pstream, there was still no mpirun. It turned out that after decomposing with decomposePar I could just write a qsub script to submit the job, where the executing line looks like

aprun -n <nr_of_processors> <solver> -parallel > log.txt
As simple as that! Now it's crunching time :D.

namCFD February 27, 2011 23:17

Hello Gijs,

I am trying to install 1.7.x on Cray XT4 but running into trouble: /usr/bin/lid cannot find lOpenFOAM. Is it possible to for you to list what you have done so I can follow to install?

I am using c-shell (csh) instead of bash and PrgEnv-gnu instead of PrgEnv-pgi. gcc is 4.4.1

I was able to install 1.7.1 but it is not working in parallel with aprun. In 1.7.1, I can use mpirun.

Thank you for your help.

gwierink February 28, 2011 01:40

Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.

namCFD March 15, 2011 19:11

Hello Gijs,

After modifying your write-up to fit my system, I got the following error while running a batch job using "aprun": aprun -n 8 icoFoam -parallel with cavity tutorial. The error message is:

icoFoam: error while loading shared libraries: cannot open shared object file: No such file or directory

Do you have any hints on how to solve this error? I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.



gwierink March 16, 2011 03:30

Hi Nam,


I tried including the directory in the LD_LIBRARY_PATH in my batch script but it is not working.
Ok, so what did you do exactly? You know where lives and you added it to LD_LIBRARY_PATH?

namCFD March 16, 2011 15:33

Hi Gijs,

Update: I contacted the system administrator and indeed does not exist in the compute node. Do you have any ideas on how to solve this issue?

First of all, thank you for your time and help.

I could not get OpenFOAM-1.7.x compiled correctly on Cray XT4 but OpenFOAM-1.7.1 is ok after using foamInstallationTest command. I don't know why.

I know that the lives in /usr/lib64 in my system. So, I did this in my batch script: export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH.

Do you think /usr/lib64 might only exist in the log-in node but not on compute nodes?


namCFD March 23, 2011 17:07

Hello Gijs,

Can you help me with this error message?

bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

From function Pstream::init(int& argc, char**& argv)
in file Pstream.C at line 73.


gwierink March 24, 2011 02:36

Hi Nam,


bool Pstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor
This looks like a run time error, is that correct? If so, did you decompose the case with decomposePar?

namCFD March 24, 2011 12:34

Hi Gijs,

I did decompose the case before submitting the case to HPC with aprun. However, we compiled OpenFOAM using openMPI that is in the ThirdParty folder. I think this is the problem because the compute nodes do not have openMPI, only MPICH2. Do you think this is the cause? We are trying to recompile with MPICH2 but as you already know, it is not a simple process. We are now trying to install OpenFOAM on Cray XE6. Do you have any pointers?


SMatt March 11, 2014 10:20


Originally Posted by gwierink (Post 297227)
Hi Nam,

I made a short report on how I compiled OpenFOAM-1.6.x on Cray XT, you can find it here. It lists all the changes etc that worked for me, hope it's helpful for you too.

I know this is an old thread, but I am struggling with getting OpenFOAM 2.1.1 to work in parallel on a Cray XE6. The links in the above post are broken.

Can anybody point me to any other threads on how to make the install work, or should I just follow the steps described above by gwierink?


All times are GMT -4. The time now is 07:03.