[fnellmeldin]

Hello. I'm new to the community and new to OpenFoam.

I need to create a script that launches OpenFoam and perform the calculations, all without user intervention.
In fact, I would like to have a script.sh which does everything and the user has only to execute this file. This is needed because we have to send jobs from HTCondor to our cluster.

Right now, to launch calculations, we do:
In bash:
openfoam4-linux # opens OpenFoam environment with Docker

In OpenFoam environment:
blockMesh
simpleFoam

But I would like to have and execute all commands, openfoam4-linux included, in one script.sh file. But, as openfoam is launched inside Docker, I don't know how to pass/execute openfoam commands from outside.

I do know that you can launch Docker detached with the -d flag, and then send commands with docker exec --name container_name commands_to_execute. But when I try this, blockMesh/simpleFoam are unknown in the normal bash environment.

Is it possible to do this? how?
An alternative is to read all of openfoam environment directly in bash, and then I would put all commands, without using Docker.

The Operative System is a fresh CentOS 7.

Thank you very much.

[wyldckat]

Quick question: I've never done this myself (use Docker in a cluster), but I need to first ask the following question: Are you planning on using more than one machine (node) on the cluster?

I ask this because if Docker is being used, then one instance of the container has to be launched on each machine that is being used, which means that those containers would have to be all pre-launched ahead of time before you can run the internal script in the master machine-container.

[cfdsolver1]

I have no experience with Docker. Is it different than running Bash shell script on HPC cluster? This is how I run my simulations using script on cluster. Maybe you can use them to find out what you need.

I first load openFoam by:

source $HOME/OpenFOAM/OpenFOAM-2.3.x/etc/bashrc

then

. $WM_PROJECT_DIR/bin/tools/RunFunctions

Finally, I can run applications by:

runApplication blockMesh

or if you need to run something parallel:

runParallel chtMultiRegionSimpleFoam

Then save it as myrun.sh for example and run it as ./myrun.sh

[wyldckat]

The detail is that Docker is somewhat of a virtualization mechanism, or in other words, it uses "containers" and each instance of a container can act as its own machine.

The problem now depends on whether you only want to use 2 or 4 cores to run the case in a single machine or if want to use multiple machines to work cooperatively.

Now, based on what you provided, I'm guessing that since you are using "runParallel" to launch the solver, then you will only be using the cores available on a single machine.

If this is the scenario you are planning on following, then do you know which installation instructions were followed for that installation? In other words, do you know if the instructions followed were these: https://openfoam.org/download/4-1-linux/

[cfdsolver1]

Well, I compiled OpenFOAM source code into my home folder as I have no root access on HPC cluster. I followed the instructions on OpenFOAMWiki.

I request 70 cores from different 5 nodes while running my cases. If I decompose my domain into 70, OpenFOAM 4.x takes the value of 70 directly from decomposeParDict while OpenFOAM 2.3.x requires you define 70 as

 runParallel <solver-name> 70

[wyldckat]

Wait, hold on, we need to retrace what you've described so far:

The HPC cluster has OpenFOAM 4.1 installed in it, but it's the Docker based version?
Which I guess was possibly the one installed as instructed here: https://openfoam.org/download/4-1-linux/

I ask this, because you mentioned that you want to use the script "openfoam4-linux", which hints at this possibility.
We need to be clear on this detail first, regarding if it's true or not, because this is very critical.



On the other note, you are aiming to use 70 cores, distributed over 5 machines... that means that "runParallel" will likely not run as you are planning on.
Are you certain that with using "runParallel" in OpenFOAM 2.3.x this worked OK in that cluster? Are you certain it didn't simply use 70 cores on a single machine?

[cfdsolver1]

Well, I have no experience on Docker and my system is not like that I guess, even though I am not sure but I assumed Docker as virtualization method and after you start the environment, I assume that you need to use everything similar with a normal PC. Isn't that correct?

About running 70 cores, for the HPC cluster I use, yes but it might be needed to include mpirun command depending on the system maybe.

[wyldckat]

Hi cfdsolver1,

Originally Posted by cfdsolver1

Well, I have no experience on Docker and my system is not like that I guess, even though I am not sure but I assumed Docker as virtualization method and after you start the environment, I assume that you need to use everything similar with a normal PC. Isn't that correct?

Yes, that's the idea, but the detail is that this new "normal PC" you go into can have it's own characteristics, such as network accesses and so on. For example, it might be able to access the Internet or local network, but it may or may not be accessible from another container.

For example, my concern here is that the 5 machines you want to use in the HPC cluster, will have an instance of a Docker container running OpenFOAM 4.1, for example:

• machine1 has a container that is running the contained machine "openfoam4-linux1"
• machine2 has a container that is running the contained machine "openfoam4-linux2"
• ...
• machine5 has a container that is running the contained machine "openfoam4-linux5"

but each instance might not be able to see the other instance, for example, "openfoam4-linux1" might not be able to see "openfoam4-linux5".


So my concern here is if the administrators of the HPC cluster have this capability configured properly or not, namely if each Docker container will be able to see the other containers. Therefore, I strongly suggest that you first contact the administrators about this; and they possibly should contact the OpenFOAM Foundation for more details on how this can be done, if they don't know yet.

Because as far as I know, when deploying an OpenFOAM installation on a cluster, it's best to install the same way that OpenFOAM 2.3.x in that cluster, namely from source code directly onto the real machines in the cluster (or a shared folder in cluster) and not into a Docker container.

Originally Posted by cfdsolver1

About running 70 cores, for the HPC cluster I use, yes but it might be needed to include mpirun command depending on the system maybe.

You might want to first confirm with the cluster administrators on how to proceed exactly. Or at the very least, what options need to be added to the mpirun command.

Because if there are no additional options needed for using mpirun, such as defining a machine list or machine file, then you can use runParallel without any further modifications.

Best regards,
Bruno

[fnellmeldin]

Hello, I am the original poster and I was asking about running OpenFoam within Docker in a Cluster. I was in vacation since the end of may, so I didn't follow the conversation.

Indeed, I had OpenFoam installed using the instructions in https://openfoam.org/download/4-1-linux/ , than means, using Docker and without compiling from Source. What I wanted to do is to launch a script within this Docker container.

As I didn't find a solution, I changed to using OpenFoam compiled by source, which is what I should have done in the first place. This way, I can do what I wanted, I don't need to use Docker and I can launch the script with all the steps as desired. This will also simplify when working with multiple machines, as mentioned in the other comments, because I guess that docker containers in different computers can't see each other.

TL;DR: I don't use OpenFoam within Docker anymore, I switched to OpenFoam compiled from source, and I can define my own scripts as desired. Problem solved (I'd say: problem avoided)

Thank you all!

[wyldckat]

Greetings fnellmeldin,

Originally Posted by fnellmeldin

Hello, I am the original poster and I was asking about running OpenFoam within Docker in a Cluster. I was in vacation since the end of may, so I didn't follow the conversation.

Many thanks for getting back to us on this!

And my apologies to both of you, for not noticing that cfdsolver1 wasn't the original poster... That's what happens when I'm in a hurry... I miss a few critical details.

Best regards,
Bruno