CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Installation

Installing OpenFOAM-1.5-dev on a cluster

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   November 16, 2010, 23:36
Default Installing OpenFOAM-1.5-dev on a cluster
  #1
ZKM
New Member
 
Join Date: Mar 2010
Posts: 12
Rep Power: 16
ZKM is on a distinguished road
Hello all,
It is a while that I am trying to install OpenFOAM-1.5-dev on a cluster without any success. I use gcc-4.3.5 by loading it as a module because system’s default gcc is 3.4.1. I set WM_COMPILER_DIR accordingly. We have HPMPI and set that in cshrc file. I install in a directory other than the standard ~ directory due to speed limitation of home. I cleaned before reinstalling. Applications are not made but foamInstallationTest does not report any problem except:
Code:
 
Checking the OpenFOAM env variables set on the LD_LIBRARY_PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory                Valid Path Crit
-------------------------------------------------------------------------------
$MPI_ARCH_PATH       /opt/hpmpi                                    yes   no  yes
-------------------------------------------------------------------------------
And its check for icoFoam apparently complains that there is no icoFoam.

I really appreciate if someone could have a look on the attached log file and advise me how to solve the problem.
Many thanks in advance
ZKM

P.S. Sorry I had to delete parts of the file due to file size restriction.
Attached Files
File Type: zip trimmedLOG.zip (92.1 KB, 7 views)

Last edited by ZKM; November 17, 2010 at 01:18.
ZKM is offline   Reply With Quote

Old   November 17, 2010, 04:59
Default
  #2
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 383
Rep Power: 18
gwierink is on a distinguished road
Hi ZKM,

On line 113 of your log it reads
Quote:
make: *** [/libPstream.so] Error 1
OpenFOAM can compile, even if PStream did not. Then, parallel runs are not possible. It may also be possible that some paths are not set correctly, so icoFoam is not found. Make sure that the MPI parths (like MPI_HOME) are set right, so that also the parallelization part of ThirdParty (metis, parmetis etc).
I have ported OpenFOAM to a large Cray cluster recently. The cluster is a bit different than yours I guess, but you may find my report on it useful. There it is explained what to change and how. Hope it helps!
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   December 25, 2010, 11:48
Default
  #3
ZKM
New Member
 
Join Date: Mar 2010
Posts: 12
Rep Power: 16
ZKM is on a distinguished road
Hello all and thanks Gijs for your help

I am still in battle for installing it. I managed to find the source of the libPstream.so problem. For some reasons that I can't understand FOAM_MPI_LIBBIN is not set inside the switch ("$WM_MPLIB") in settings.csh. I set it manually again:

Code:
setenv FOAM_MPI_LIBBIN $FOAM_LIBBIN/hpmpi
It solves this problem but I still have this typical errors/warnings for all solvers:
Code:
+ cd applications
+ ./Allwmake
+ cd solvers
+ wmake all
make[2]: Nothing to be done for `application'.
make[2]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[2]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[3]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
make[4]: Nothing to be done for `application'.
Making dependency list for source file conjugateHeatFoam.C
Making dependency list for source file potentialFoam.C
SOURCE=potentialFoam.C ;  g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/potentialFoam.o
SOURCE=conjugateHeatFoam.C ;  g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/coupledMatrix/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/conjugateHeatFoam.o
g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/coupledMatrix/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/conjugateHeatFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lcoupledLduMatrix -lfiniteVolume -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/conjugateHeatFoam
g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/potentialFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lfiniteVolume -llduSolvers -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/potentialFoam
/usr/bin/ld: warning: libPstream.so, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmpi.so.1, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so, not found (try using -rpath or -rpath-link)
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Finalize'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Buffer_detach'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_setkind_np'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_create'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_rdlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Bsend'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Probe'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Send'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_unlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Test'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Allreduce'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Buffer_attach'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Comm_size'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Recv'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Isend'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Get_count'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Abort'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Waitall'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Get_processor_name'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_wrlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Comm_rank'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_settype'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `MPI_Irecv'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_sum'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlockattr_destroy'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_trywrlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_rwlock_tryrdlock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutexattr_init'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_byte'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_double'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_mutex_trylock'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_packed'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so: undefined reference to `hpmp_comm_world'
/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so: undefined reference to `pthread_join'
collect2: ld returned 1 exit status
make[2]: *** [/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/conjugateHeatFoam] Error 1
make[1]: *** [conjugateHeatFoam] Error 2
make[1]: Target `application' not remade because of errors.
make: *** [conjugate] Error 2
/usr/bin/ld: warning: libPstream.so, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmpi.so.1, needed by /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so, not found (try using -rpath or -rpath-link)
I have no idea how to use -rpath or -rpath-link as suggested in the warning message.

I found out that libmpi.so.1 is located in two locations /opt/hpmpi/lib/linux_amd64 and /opt/hpmpi/lib/linux_amd64/nomalloc/

I am going to do something like this manually:

Code:
setenv LD_LIBRARY_PATH /opt/hpmpi/lib/linux_amd64:$LD_LIBRARY_PATH
but I am not sure which libmpi.so.1 is required.

As for the other warning, that I think ends in those errors, again I don't know which libPstream.so is needed (in lib/linux64GccDPOpt/dummy or in lib/linux64GccDPOpt/hpmpi) and how to tell it to use from that path.

Our admin advised me to add -lpthread after the "-ldl" in:
Code:
g++ -m64 -Dlinux64 -DDP -DFOAM_DEV_REVISION_NUMBER=1861 -Wall -Wno-strict-aliasing -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-40 -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/finiteVolume/lnInclude -IlnInclude -I. -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OpenFOAM/lnInclude -I/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/src/OSspecific/Unix/lnInclude   -fPIC Make/linux64GccDPOpt/potentialFoam.o -L/sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt \
     -lfiniteVolume -llduSolvers -lOpenFOAM -liberty -ldl /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/hpmpi/libPstream.so   -lm -o /sfs/nobackup/z/z_kesha/OpenFOAM/OpenFOAM-1.5-dev/applications/bin/linux64GccDPOpt/potentialFoam
because he thinks that the pthread library was not linked in, but again I don't know where this command is read from to edit it.

Any help and comments are appreciated.

Last edited by ZKM; December 25, 2010 at 14:46.
ZKM is offline   Reply With Quote

Old   December 25, 2010, 15:06
Default
  #4
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 383
Rep Power: 18
gwierink is on a distinguished road
Hi ZKM,

Probably it is best to create your own wmake rules for your cluster compiler. What kind of cluster are you building on actually? I built OpenFOAM on different clusters and needed to do that. I guess you could export every rules, as you describe, but that'll become a mess quite easily I'm afraid. Here you can find a manual I wrote. It's for OpenFOAM-1.6.x on Cray cluster, but the general recipe should be useful.
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   December 25, 2010, 16:59
Default
  #5
ZKM
New Member
 
Join Date: Mar 2010
Posts: 12
Rep Power: 16
ZKM is on a distinguished road
So I am not the only one who works on Christmas

I have already read your document but couldn't solve the problem. Here is some information about the cluster:
http://users.encs.concordia.ca/~cirrus/1.html

Yes very true and I must implement all changes in wmake rules once I really understand how this thing works.

How threading is done in OpenFoam? Is it through Pstream? If not I guess I should follow that advice to link pthread by putting -lpthread after -ldl in the "general" rule file (is the file correct?)

ZKM
ZKM is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
[OpenFOAM] ParaFoam instalation in OpenFoam 1.5 dev titio ParaView 2 July 25, 2010 00:24
OpenFOAM 1.5 dev LVDH OpenFOAM 98 May 5, 2010 18:01
[OpenFOAM] Paraview/Parafoam in OpenFoam 1.5 dev titio ParaView 0 December 9, 2009 13:13
[OpenFOAM] Paraview/Parafoam in OpenFoam 1.5 dev titio ParaView 0 December 9, 2009 13:12
Problem installing OpenFOAM 1.5 installation on RHEL 4. vwsj84 OpenFOAM Installation 4 April 23, 2009 05:48


All times are GMT -4. The time now is 16:43.