CFD Online Discussion Forums

Notes about running OpenFOAM in parallel

Do you have any notes on how to use simple decomposition for parallel processing. I am running on a dual hexacore (12 core) machine, and no matter what I try, my decomposeParDict in sloshingtank2D just wont work, and I get errors something like:
Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod hierarchical

--> FOAM FATAL ERROR:
Wrong number of processor divisions in geomDecomp:
Number of domains : 12
Wanted decomposition : (4 2 2)

From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
in file geomDecomp/geomDecomp.C at line 50.

Any suggestions? why wont (6 6 1) work? Isnt that 12 sub domains? I look forward to your reply.
:

Posted July 17, 2013 at 23:43 by musahossein musahossein is offline

	ParaView 3.12.0 SuperBuild on OpenFOAM It works. Yes. This works. Thank you very much Bruno. I guess this version 2.2.0 is a stable version. Best Suranga.
	Posted May 28, 2013 at 22:30 by sdharmar

	ParaView 3.12.0 SuperBuild on OpenFOAM Hi Suranga, Try following these instructions: http://openfoamwiki.net/index.php/In...u#Ubuntu_12.04 The instructions are meant for Ubuntu 12.04, but Debian Wheezy should be very similar. Best regards, Bruno
	Posted May 27, 2013 at 16:36 by wyldckat

ParaView 3.12.0 SuperBuild on OpenFOAM

Hi Bruno,
Thank you very much for trying to help me.I am using Debian/Linux 7.0 wheezy amd64. I was keep on trying to build paraview. Finally I executed ./makeparaview just to see whether it works. Then I did a icoFoam tutorial and typed paraFoam on the terminal and it gave me following.

Code:

FATAL ERROR: ParaView reader module libraries do not exist

Please build the reader module before continuing:
cd $FOAM_UTILITIES/postProcessing/graphics/PV3Readers
./Allwclean
./Allwmake

Then I navigated to the given PV3Readers directory and executed

Code:

/Allwclean

and

Code:

./Allwmake.

Then it gave me following errors at the end.

Code:

CMake Error at CMakeLists.txt:12 (FIND_PACKAGE):
  Could not find module FindParaView.cmake or a configuration file for
  package ParaView.

  Adjust CMAKE_MODULE_PATH to find FindParaView.cmake or set ParaView_DIR to
  the directory containing a CMake configuration file for ParaView.  The file
  will have one of the following names:

    ParaViewConfig.cmake
    paraview-config.cmake



CMake Error at CMakeLists.txt:13 (INCLUDE):
  include called with wrong number of arguments.  Include only takes one
  file.


CMake Error at CMakeLists.txt:43 (QT4_WRAP_CPP):
  Unknown CMake command "QT4_WRAP_CPP".


-- Configuring incomplete, errors occurred!
+ make
make: *** No targets specified and no makefile found.  Stop.

I tried a lot an failed every instance. Can you find any clue.

Regards,
Suranga.

Posted May 27, 2013 at 01:03 by sdharmar sdharmar is offline

	ParaView 3.12.0 SuperBuild on OpenFOAM Hi Suranga, I have not been able to reproduce the same error you've gotten. I've used Ubuntu 12.04 x86_64 without any problems. Which specific Linux distribution and version are you using? Best regards, Bruno
	Posted May 26, 2013 at 10:02 by wyldckat

ParaView 3.12.0 SuperBuild on OpenFOAM

Thanks Bruno,
Actually I found this modification from you another post and did. It got me through this error but gave me another.
[/CODE]
make[5]: *** [src/thread/src/CMakeFiles/boost_thread-mt-shared.dir/pthread/once.cpp.o] Error 1
make[5]: *** [src/thread/src/CMakeFiles/boost_thread-mt-static.dir/pthread/once.cpp.o] Error 1
make[4]: *** [src/thread/src/CMakeFiles/boost_thread-mt-shared.dir/all] Error 2
make[4]: *** [src/thread/src/CMakeFiles/boost_thread-mt-static.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [Boost-prefix/src/Boost-stamp/Boost-build] Error 2
make[1]: *** [CMakeFiles/Boost.dir/all] Error 2
make: *** [all] Error 2
[/code]

What can I do for this. Please help me.
Suranga.

Posted May 23, 2013 at 20:21 by sdharmar sdharmar is offline

ParaView 3.12.0 SuperBuild on OpenFOAM

Hi Suranga,

Two weeks? You could have said something sooner...

I haven't tried these instructions in quite a while now... but the problem you're seeing is possibly because "silo-4.8-bsd.tar.gz" was changed once again on the original website.
The trick is to edit the file "SuperBuild/PackageVersions.cmake", find the this block of code:

Code:

set(SILO_MAJOR 4)
set(SILO_MINOR 8)
set(SILO_PATCH 0)
set(SILO_VERSION ${SILO_MAJOR}.${SILO_MINOR}.${SILO_PATCH})
set(SILO_URL http://www.paraview.org/files/v3.14/dependencies/)
set(SILO_GZ silo-${SILO_MAJOR}.${SILO_MINOR}-bsd.tar.gz)
set(SILO_MD5 03e27c977f34dc6e9a5f3864153c24fe)

And replace the last line with this:

Code:

set(SILO_MD5 d864e383f25b5b047b98aa2d5562d379)

Best regards,
Bruno

Posted May 23, 2013 at 19:00 by wyldckat wyldckat is offline

ParaView 3.12.0 SuperBuild on OpenFOAM

I am sorry for posting on a old post. But I am kind of desperate having been tried to install paraview for two weeks. I followed the same steps described in the very first post of the thread by Bruno. But finally running make -j 4 for more than 5 or 6 times I get the same massage.CMake Error at Silo-prefix/src/Silo-stamp/download-Silo.cmake:6 (file):
file DOWNLOAD MD5 mismatch

for file: [/home/suranga/OpenFOAM/ThirdParty-2.1.x/build/linux64Gcc/SuperBuild/silo-4.8-bsd.tar.gz]
expected MD5 sum: [03e27c977f34dc6e9a5f3864153c24fe]
actual MD5 sum: [d864e383f25b5b047b98aa2d5562d379]

make[2]: *** [Silo-prefix/src/Silo-stamp/Silo-download] Error 1
make[1]: *** [CMakeFiles/Silo.dir/all] Error 2
make: *** [all] Error 2

What would be the reason for this and how may I fix this.

Thanks
Suranga.

Posted May 23, 2013 at 01:22 by sdharmar sdharmar is offline

	Advanced tips for working with the OpenFOAM shell environment Greetings Mohsen, Sorry for the late reply. I think the problem is that you left a space between the = and ': Quote: Code: alias of211= 'source /opt/OpenFOAM-2.1.1/etc/bashrc' When there shouldn't be a space after the equal sign: Code: alias of211='source /opt/OpenFOAM-2.1.1/etc/bashrc' Best regards, Bruno
	Posted November 26, 2012 at 15:13 by wyldckat

Advanced tips for working with the OpenFOAM shell environment

Hi, I have OpenFoam2.1.1 installed on my system already. And now I need to have OpenFoam1.5 as well. So now I have this line in my .bashrc: source /opt/OpenFOAM-2.1.1/etc/bashrc
So should I now just simply comment that line out and add the below line?
alias of211= 'source /opt/OpenFOAM-2.1.1/etc/bashrc'
I did that but when I type of211 in my terminal, I get the error "command not found"
please let me know what I'm missing,

Thanks
Mohsen

Posted November 15, 2012 at 18:26 by mhsn mhsn is offline

Advanced tips for working with the OpenFOAM shell environment

Hi Giuliano,

Well, I did write on the blog post that it isn't complete yet...

Nonetheless, OpenFOAM can be installed for all users (in "/opt" or "/usr/local" or anything similar) or be installed for a single user (usually "$HOME/OpenFOAM").
If you look at the instructions for building from source and git repository, you'll see that "$HOME/OpenFOAM" is standard installation path for these instructions!

This alias methodology (FYI: it was not me who discovered it

) can be used for any installations of OpenFOAM. For example, you can easily have this:

Code:

alias of20x='. $HOME/OpenFOAM/OpenFOAM-2.0.x/etc/bashrc'
alias of211='source /opt/openfoam211/etc/bashrc'
alias of21x='source $HOME/OpenFOAM/OpenFOAM-2.1.x/etc/bashrc'
alias of16ext='. $HOME/OpenFOAM/OpenFOAM-1.6-ext/etc/bashrc'

Note: The source command and the dot command are basically identical.

As for where to place these aliases:

Do not confuse these two files:
- "OpenFOAM-2.1.x/etc/bashrc" - this file has the settings for OpenFOAM.
- "~/.bashrc" is the personal file for setting up the shell environment.
You do not have to define these aliases in "~/.bashrc", because there are dedicated files for this, as you can see from the source code of that file. Examples:
- Ubuntu has a dedicated user file for aliases: "~/.bash_aliases"
- openSUSE has this one: "~/.alias"
Both are called from the user's "~/.bashrc" file.
And yes, the idea is to replace the conventional line, such as this one:
Code:
```
source /opt/openfoam211/etc/bashrc
```
And replace that line for the various aliases you wish!

This way you don't need a user per version of OpenFOAM!

As for running in parallel, the trick is to use foamJob, which relies on foamExec to setup the correct environment on the remote machines.

As for system-wide installations, the advice is basically this:

Build with a normal user account and follow the desired building instructions.
When finished, go into root mode, copy the whole installation to the desired global folder, such as "/opt/OpenFOAM".
Then change the permissions on this whole folder, to avoid unintentional code/binary modifications.

Edit the file "OpenFOAM-*/etc/bashrc" and find this big block of unmistakable code:

Quote:

Code:

################################################################################
# USER EDITABLE PART: Changes made here may be lost with the next upgrade
#
# either set $FOAM_INST_DIR before sourcing this file or set
# 'foamInstall' below to where OpenFOAM is installed
#
# Location of the OpenFOAM installation
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
foamInstall=$HOME/$WM_PROJECT
# foamInstall=~$WM_PROJECT
# foamInstall=/opt/$WM_PROJECT
# foamInstall=/usr/local/$WM_PROJECT
#
# END OF (NORMAL) USER EDITABLE PART
################################################################################

I think it explains for itself

Switch back to normal user and check if everything is working as intended.

You can easily create a global files with aliases for all users to use, such as "/opt/OpenFOAM/.ofaliases" and tell users to add the following line to their own "~/.bashrc" file:

Code:

source /opt/OpenFOAM/.ofaliases

Best regards,
Bruno

Posted October 14, 2012 at 15:38 by wyldckat wyldckat is offline

Advanced tips for working with the OpenFOAM shell environment

Hi Bruno.

I would propose a change:
I would make the alias point to the global openfoam bashrc, and not the local one
When you install openfoam (I means SGI OF 2.1.1) the installer suggest to source your .bashrc with the "global" bashrc, not the local one
. /opt/openfoam211/etc/bashrc

http://www.openfoam.org/download/ubuntu.php...
even for suse users :-)
http://www.openfoam.org/download/suse.php

(even more, my special account "cfduser" doesn't even have the ~/etc directory...)

------------------------------------------
Next my two questions:

1) your main idea is to REMOVE the sourcing of the OF bashrc, and to SUBSTITUTE it with the alias. Right ?

2) I had the idea to create two different user (say cfduser2.1.1. and cfduser 1.7.1) and install two different bashrc in the two.
Your trick seems clearly better, because can keep the same user, but
does even manage the different programs like
-paraview
-snappy
-...

that the different distribution may install ?

Posted October 14, 2012 at 14:59 by Giuliano69 Giuliano69 is offline

	OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs Quote: But, why are you using "-ffast-math"? AFAIK, that option is hated in the CFD world, because it doesn't offer full precision nor reproducibility! it depends on task, because it only does not guarantee precision in some cases
	Posted August 1, 2012 at 16:41 by SergeAS

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

Many thanks for the info! I did think that the "-pipe" option was something for optimizing the way cache was handled in the core... I should have read the manual

But, why are you using "-ffast-math"? AFAIK, that option is hated in the CFD world, because it doesn't offer full precision nor reproducibility!

edit: OpenMP is pointless in OpenFOAM, because it doesn't take advantage of multi-threading capabilities, nor is OpenFOAM thread-safe in some cases!

Posted August 1, 2012 at 16:16 by wyldckat wyldckat is offline

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

In my solver I'm using next options set for gcc

Code:

-O3 -ffast-math  -funroll-loops -fprefetch-loop-arrays -frerun-loop-opt -ffloat-store  -frerun-cse-after-loop -fschedule-insns2 -fno-strength-reduce -fexpensive-optimizations -fcse-follow-jumps -falign-loops=32 -falign-jumps=32 -falign-functions=32 -falign-labels=32 -fcse-skip-blocks  -ftree-vectorize  -msse3 -msse2 -msse -mmmx -m3dnow -mfpmath=sse -mtune=opteron

but it seems for current version of solver - GCC have no real chance against icc (in serial version)

PS: in case of OpenMP solver both compilers has horrible performance/scaling for NUMA systems on Opterons but gcc 4.5.1 a little bit faster then icc 10.1

in case of dual head Operon285 in 16Gb node (2x2 core) on same test (6 m nodes - ~10 GB overal task size) the results is next:

icc 10.1 OpenMP - 8.15 sec (~29% parallel effectivity on 4 cores )
gcc 4.5.1 - 7.95 sec (~47 % parallel effectivity on 4 cores)

For NUMA systems is more appropriate MPI as programming model (my test task has ~41% parallel effectivity ... on 54 cores in case of MPI )

PPS: One small remark
option -pipe speedup compiling of OF but not the execution of the solver code

Quote:

-pipe'
Use pipes rather than temporary files for communication between the
various stages of compilation.

so it is well loved in gentoo

Posted August 1, 2012 at 15:16 by SergeAS SergeAS is offline

Updated August 1, 2012 at 16:00 by SergeAS

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

I've finally managed to complete the benchmarks! That "-msse3" option did bring good results! But I then went out to get the flags for GCC + 1055T and it did even better

edit: I updated the last part of the blog post

For your Opteron 285, looks like these are the perfect gcc options:

Quote:

Originally Posted by http://en.gentoo-wiki.com/wiki/Safe_Cflags/AMD#2xx.2F8xx_Opteron

-march=opteron -O2 -pipe

PS: really nice blog posts you've got about performance comparisons!!

Posted August 1, 2012 at 05:37 by wyldckat wyldckat is offline

Updated August 1, 2012 at 05:38 by wyldckat (see "edit:")

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

It would be interesting to see comparative benchmarks gcc and icc on OF.

PS: In my own density-based solver ( for compressible reacting flows from subsonic to hypersonic) icc beat gcc by about 10% but it is quite old compilers (icc 10.1 vs gcc 4.1) and processors - (AMD Opteron285 and Intel Xeon 5120)

UPD: I spent a little test for my current problem for the "pure" performance of the solver (serial version), for 3 different compilers: gcc 4.1.0, gcc 4.5.1 and icc 10.1 and got the following results (one time step)

gcc 4.1 - 14.7 seconds
gcc 4.5.1 - 14.95 sec
icc 10.1 - 9.45 seconds

icc in this case produces code that is faster by more than 30%

(The test was performed on AMD Opteron285)

Posted July 28, 2012 at 15:08 by SergeAS SergeAS is offline

Updated July 29, 2012 at 07:59 by SergeAS

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

How did I not see this before... In the last link I posted, there is this other link: http://developer.amd.com/Assets/Comp...f-61004100.pdf (edit: the AMD website is being revamped and the PDF got lost... best to Google for "CompilerOptQuickRef-61004100.pdf")
There is indicated to use the option "-msse3" (and to avoid "-ax"), which in Icc 12.1 is the same as the now deprecated option "-oX" you've written about!

My machine is currently building another set of OpenFOAM 2.1.x with this option and then I'll run some more tests to compare performance times! If it fares well, it's a good idea to propose to include in OpenFOAM the crazy "WM_COMPILER" option "IccAMD"

Serge, many thanks for the info! Although I can almost bet that Gcc 4.6 can still outrun Icc

Posted July 28, 2012 at 13:01 by wyldckat wyldckat is offline

Updated May 18, 2014 at 12:46 by wyldckat (see "edit:")

OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs

AFAIK since icc 10 or 10.1 has special option -xO

Code:

-x<codes>  generate specialized code to run exclusively on processors
           indicated by <codes> as described below
    W  Intel Pentium 4 and compatible Intel processors
    P  Intel(R) Core(TM) processor family with Streaming SIMD
       Extensions 3 (SSE3) instruction support
    T  Intel(R) Core(TM)2 processor family with SSSE3
    O  Intel(R) Core(TM) processor family.  Code is expected to run properly
       on any processor that supports SSE3, SSE2 and SSE instruction sets
    S  Future Intel processors supporting SSE4 Vectorizing Compiler and
       Media Accelerator instructions

For several years I have been using this icc (v10.1) option for my code on my old cluster with AMD Opteron285

Code:

icc --version
icc (ICC) 10.1 20080801
Copyright (C) 1985-2008 Intel Corporation.  All rights reserved.

Posted July 28, 2012 at 09:27 by SergeAS SergeAS is offline

	OpenFOAM with ICC - Intel's ICC is "allergic" to AMD CPUs Good point, I forgot to mention in the blog post. edit: I've added it now. Nonetheless, this issue with ICC and AMD processors has several years now, as you can see from the references at Wikipedia And I don't think they (intel) have any plans to change this point of view.
	Posted July 28, 2012 at 04:10 by wyldckat Updated July 28, 2012 at 04:10 by wyldckat (see "edit:")

Page 3 of 7

Last »

Archive
< May 2024
Su	Mo	Tu	We	Th	Fr	Sa
21	22	23	24	25	26	27
28	29	30	1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31	1

Comments

It works.