[seboxx]

Hi there,

I have found some stuff on the forum and also using google.
But so far I couldn't see any proper description of how to get a static executable...

Did anyone have any success?
I am just thinking it would be generally very nice to have one single executable even if it is some mbs large.
I am interested in that in order to run parallel computation on a cluster where the nodes have no OpenFOAM package installed.

Best regards,

sebastian

[eugene]

So far the best results we have had is with a tool called "Magic Ermine". There are still some issues with MPI though and it is not free.

Unfortunately, there doesn't seem to be an easy way to do this.

[wyldckat]

Greetings to all!

@Sebastian: I've been accompanying your posts on another thread, namely http://www.cfd-online.com/Forums/ope...xe-option.html. It's been a year since my last posts on this subject.
OpenFOAM 2.0 has evolved in this direction to provide a quicker way to create static binaries, but it's only ready half way through. My experience on this subject has also increased a bit, but for now I'm only able to provide an incomplete hack: https://github.com/wyldckat/OpenFOAM...staticBuilding

Basically apply that patch and then run AllwmakeStatic instead of Allwmake.

Best regards,
Bruno

[seboxx]

Hi Bruno,

thank you for your quick reply.
I see that it is still not trivial to get static binaries...
I will have a look at your hack. It seems to be messing with the whole OpenFOAM installation doesn't it? So I better prepare a setup that is independant from my current workspace.

[wyldckat]

Hi Sebastian,

I've tried to keep the hack as contained as possible to keep the dynamic version untainted, but I haven't tested if the dynamic version still builds as intended. So the answer to your question is: yes, you better keep this hacked version on a different work folder.

I've just updated my hacking on the branch I've posted before: https://github.com/wyldckat/OpenFOAM...staticBuilding

It's still a somewhat of an ugly hack, but now all applications build statically. I haven't tested executing them to confirm if they work as intended. They all link well, so I expect that they work!
If some of the binaries don't work, then the only other way is to build first the dynamic version and then the static version, based on the library link list provided by the dynamic version.

Let me know if this is working as intended or not, since I don't know when I'll get the chance to do some testing of my own.

I failed to mention on the previous post that this hack implies that both your cluster and workstation already have libScotch and a system MPI already installed and properly configured for building with OpenFOAM; therefore, these two libraries, along with the other system libraries, will be the only ones that are dynamically linked to the static OpenFOAM binaries.

Now, all that is left, is to:

• Give this ugly hack a plastic operation...
• Test, test, test...
• Do a bit of benchmarking, just for the sake of figuring out how much additional performance can be pulled from this.

Best regards,
Bruno

[seboxx]

Hi Bruno,

I really appreciate your comments.
One thing I am considering at the moment...
I want to run my solver on a cluster, managed by condor, that has (up to now) no OpenFOAM package installed on the nodes.

What I am looking for is the easiest way to run my simulations on that cluster.

You have posted in another post:
http://www.cfd-online.com/Forums/ope...xe-option.html
that the easiest way might be to ship the solver binary along with the dependant libraries found by

ldd mySolver

to the node.

I have done exactly that and also shipped the libfiniteVolume.so but I get this in the error logs:

error while loading shared libraries: libfiniteVolume.so: cannot open shared object file: No such file or directory

Can you point me in a direction to get this solved? Is the problem some environment variables?

I feel a bit sorry for not having tested your hack so far. But working on an actual master thesis, with a time-limit, and where the physics is in the foreground makes me look for easy solutions w.r.t. to the computations

Actually I don't need to decompose my domain to different cpus, I need to run a system on reasonably simple geometries with a lot of different sets of parameters. So one stand-alone simulation on a node would do...

If I get it to work on the condor cluster without having OpenFOAM installed to all the nodes, I promise to participate in writing a wiki article if desired

Sebastian

[wyldckat]

Hi Sebastian,

OK, here's an example:

1. You must build OpenFOAM with the system's MPI, not with the custom OpenMPI, otherwise this will fail big time!
2. Go to the case you want to run; e.g.:

   cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavityGrade

3. Copy the necessary binary; e.g. blockMesh:

   cp `which blockMesh` .

4. Copy the necessary libraries for running the binary; e.g. blockMesh:

   ldd `which blockMesh` | grep OpenFOAM | grep -v libmpi | sed -e 's=.*\('$WM_PROJECT_INST_DIR'.*\) .*=\1=' | xargs -I {} cp {} .

5. Copy OpenFOAM's etc folder:

   cp -r $WM_PROJECT_DIR/etc .

6. Now, somehow you must set in the job description for the cluster the following commands or analogous:

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
   export WM_PROJECT_DIR=.

   These commands will add the relative current folder to the library search paths as well as defining the necessary variable for OpenFOAM to work.

I only tested these out on a single machine, where:

• one terminal had a working OpenFOAM environment;
• while the other terminal did not have a working OpenFOAM environment, only running the commands described in point 6.

As for having a working OpenFOAM environment only when I need it, see this blog post: Advanced tips for working with the OpenFOAM shell environment - point 3
edit: As for more information about running in parallel, here is a compilation of information on the subject: Notes about running OpenFOAM in parallel

As for Condor Cluster system, I've never used it and I don't have access to one, so I think this is as far as I can help you

Best regards and good luck!
Bruno

[seboxx]

Hey Bruno,

it is working nicely!
I am very happy!
It really runs nicely on all nodes of the condor cluster with just my local installation of OpenFOAM having recompiled it with the right MPI options...

You really helped me out a lot here and I will surely mention you in my Acknowledgement.

[wyldckat]

Hi Sebastian,

Originally Posted by seboxx

You really helped me out a lot here and I will surely mention you in my Acknowledgement.

You're very welcome! I'm glad it worked as well as intended. And many thanks for the Acknowledgement

Although I feel a bit bad that I didn't manage to get around to this months ago.. oh well, can't do everything all at once .

Ah, one final note: sometimes there is also a need to include in the job package the libraries that are summoned by the controlDict for the case you're running! Such an example would be the somewhat famous libforces.so library.

Best regards and good luck!
Bruno

[seboxx]

Hi Bruno,

I am running into another Problem.
It used to work once, but now it seems like it doesnt work anymore.
Executing everything locally works as good as before. But on the cluster it stops after having executed the first time step each run...

What I get is the following error message:

--> FOAM FATAL IO ERROR:
wrong token type - expected Scalar, found on line 3 the word 'nan'

file: /users/boie/simulations/flat_slope0.00005beta0.06/system/data::solverPerformance::u at line 3.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

FOAM exiting

and this in the log files
Time = 0.0001

GAMG: Solving for u, Initial residual = 0.000118434, Final residual = 1.68319e-16, No Iterations 1
diagonal: Solving for v, Initial residual = 0, Final residual = 0, No Iterations 0
GAMG: Solving for u, Initial residual = 1, Final residual = 1.06043e-17, No Iterations 1
diagonal: Solving for v, Initial residual = 0, Final residual = 0, No Iterations 0
wavesize= 0ExecutionTime = 0.22 s ClockTime = 0 s

Time = 0.0002

GAMG: Solving for u, Initial residual = 1, Final residual = nan, No Iterations 1000
diagonal: Solving for v, Initial residual = 0, Final residual = 0, No Iterations 0
GAMG: Solving for u, Initial residual = nan, Final residual = nan, No Iterations 1000

I find surprising that the Final residual in the second timestep is nan...

As mentioned this problem only occurs when running on cluster.
Any ideas?

EDIT:

I have done a bit more testing. And strangely I get different results when running locally and on the cluster. I have executed one small time step and printed out the results, once from cluster calculation and once locally.
When running locally my scalarField has mostly values of the order e-100 over the whole domain, which is reasonable since I only have a small fraction of the field not equal to zero and have a diffusion process involved. So this seems fine...

However once I compute it on the cluster I get a value of -1.80998e+97 on each grid point, even the same value. No wonder my calculations diverge immediately. Only I cant see why...

EDIT2:

I have also tried to run the solver on another local machine without openfoam, the result is similar. So I think it is some problem that already appears when compiling the solver... it is executable on any machine but doesnt give any decent results

EDIT3:

I found the root of the problem. I was declaring a scalar with double S;
and assigning a value and later using it in one of the equations. This seems not to be appropriate way of coding in OpenFOAM. Using a class out of the openfoam package instead of a simple variable declaration solved the problem! I dont know why it wouldnt handle the double properly... but anyway its working ;-)
Sorry for all the editing and posting ;-)

[wyldckat]

Hi Sebastian,

Sorry for the late reply, but I'm really glad that you managed to sort things out on your own! And I do prefer that people edit their posts, instead of making new ones... unless it's really worth the a new post

Since you were seeing "nan" in results, I was going to suggest that you would check the following variables defined in OpenFOAM's "etc/bashrc":

#- Floating-point signal handling:
#    set or unset
export FOAM_SIGFPE=

#- memory initialisation:
#    set or unset
#export FOAM_SETNAN=

When these are defined in the environment, they do what the comments talk about. When they are not defined, which is the situation for your when running in the cluster, these would be the reasons why things were different! Very different indeed!


But that "double S" declaration was really a bad idea OpenFOAM likes and demands that things are done the right way, or else... there will be consequences!

Best regards and good luck!
Bruno

[Rebecca513]

Hello,

I am trying to run OF on a cluster, and ran into the same error messages. I wonder whether you have solved the issue.

Thank you~

Best,

Hang

[seboxx]

Hey Hang,

yes using a cluster without openfoam on the individual nodes eventually worked for me. What error message do you get?

best,

seboxx

[Rebecca513]

Hi seboxx,

The error message I got is very similar to the one you posted:


--> FOAM FATAL IO ERROR:
wrong token type - expected Scalar, found on line 3 the word 'nan'

file: /scratch/gpfs/hangdeng/FOAM_Run/fracAfter/test2/system/data::solverPerform
ance: at line 3.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

FOAM exiting

Thank you~

Best,
Hang

[seboxx]

Well for me the problem was a poorly implemented additional function.

I don't know where your calculations go wrong. Did you implement functions to the solver? If so maybe try to comment them out and see whether the same thing happens.

Does the simulation produce good results when you run them on a single machine?

[Rebecca513]

The solver actually works for one of the tutorial case (both parallel and non-parallel), but failed my mesh in both parallel and non-parallel cases.

So, I don't think it is because of the additional function. I guess there might be something wrong with my mesh, I will look into that~

Thank you~

Best,

Hang

[mr-albert]

@ Rebecca
Hi how are you?
i have problem similar yours

this is my Error
--> FOAM FATAL IO ERROR:
wrong token type - expected Scalar, found on line 3 the word 'nan'

file: /home/mohammadreza/Desktop/periodicSolitary/system/data::solverPerformance:corr at line 3.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

FOAM exiting


do you solve your problems?
could you help me?

king regard
mr-albert