Slow OF job on cluster w/ externalCoupled BCs

Gerry Kan · July 5, 2018, 03:28

Howdy folks:

I am wondering if you have also seen this, or if you have any ideas how I should proceed...

Currently I am exploring the possibility of using the externalCoupled boundary conditions to link OpenFOAM with another solver. As first test I modified the tutorial case (heatTransfer/buoyantSimpleFoam/externalCoupledCavity) and run this locally. This ran without problems and I was satisfied with the performance.

Then I proceeded to testing this on a cluster, to which I submitted the OF job. The dummy solver (externalSolver in the tutorial) still runs locally, and the data exchange takes place in the default directory (comms). But this time I notice a very long lag (~ 30 seconds) between time steps.

Something tells me that I might have overlooked something when I am setting this case up on the cluster environment. A possible solution *could* be writing directly to the staging directory, but again I don't know enough to qualify my hunch.

I am hoping that there might be others who have successfully done this and point me in the right direction.

NOTE: I am also contemplating implementing the data exchange completely in OF. However, I would like to reserve this option until it is absolutely clear that it is the only way to proceed.

Thanks in advance, Gerry.

Gerry Kan · July 9, 2018, 09:00

After a bit of digging, I realized that when my OpenFOAM job and my externalSolver job are started on difference servers but on the same file system, somehow my both jobs will wait for (exactly) 30 seconds before proceeding to the next iteration.

So I think the issue is NFS related?

Is there a way to reduce this time (without getting our sysadmin going crazy on me)?

OR, is there a way to synchronize the two processes?

Thanks, Gerry.

P.S. - I saw this thread on the CFD-Online forum and tried the options, but it did not change the 30-second lag.

Gerry Kan · July 11, 2018, 02:40

It looks like it is definitely an NFS lag thing. What I find strange is that:

a) In externalMixedFvPatchField::createLockFile ( ), a flush() is called immediately after creation of said lock file. So I don't know why it takes another 30 seconds to get picked up by the other process through the NFS. But then again, OF looks for the lock file from the other process anyways.

b) On my other process (externalSolver in the tutorial) I made a call to sync after touching the lock file. Still, it took OF about 10 - 20 seconds to pick it up. So I guess the syncing worked, but not as much improvement as I wanted.

In the meantime I am just going to change the BC definition and have OF skip the lock file checking. At the moment I am not worried about generating the BCs so it is all good.

Gerry.

July 5, 2018, 03:28	Slow OF job on cluster w/ externalCoupled BCs	#1
Gerry Kan Senior Member Gerry Kan Join Date: May 2016 Posts: 347 Rep Power: 10	Howdy folks: I am wondering if you have also seen this, or if you have any ideas how I should proceed... Currently I am exploring the possibility of using the externalCoupled boundary conditions to link OpenFOAM with another solver. As first test I modified the tutorial case (heatTransfer/buoyantSimpleFoam/externalCoupledCavity) and run this locally. This ran without problems and I was satisfied with the performance. Then I proceeded to testing this on a cluster, to which I submitted the OF job. The dummy solver (externalSolver in the tutorial) still runs locally, and the data exchange takes place in the default directory (comms). But this time I notice a very long lag (~ 30 seconds) between time steps. Something tells me that I might have overlooked something when I am setting this case up on the cluster environment. A possible solution could be writing directly to the staging directory, but again I don't know enough to qualify my hunch. I am hoping that there might be others who have successfully done this and point me in the right direction. NOTE: I am also contemplating implementing the data exchange completely in OF. However, I would like to reserve this option until it is absolutely clear that it is the only way to proceed. Thanks in advance, Gerry.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
bash script for pseudo-parallel usage of reconstructPar	kwardle	OpenFOAM Post-Processing	41	August 23, 2023 02:48
Submit a cfd job with shell on HPC cluster	matheusmonjon	Hardware	1	December 19, 2017 11:32
Slow Cases using HPC and PBS job scheduler	Alhasan	OpenFOAM Running, Solving & CFD	5	February 1, 2015 11:07
OpenFOAM 2.2.2 on a cluster as a job FlexLexer error	Artur	OpenFOAM Installation	4	December 8, 2013 15:28
Very Very SLOW interFoam on Cluster	ovie	OpenFOAM	3	May 31, 2010 04:06

July 9, 2018, 09:00		#2
Gerry Kan Senior Member Gerry Kan Join Date: May 2016 Posts: 347 Rep Power: 10	After a bit of digging, I realized that when my OpenFOAM job and my externalSolver job are started on difference servers but on the same file system, somehow my both jobs will wait for (exactly) 30 seconds before proceeding to the next iteration. So I think the issue is NFS related? Is there a way to reduce this time (without getting our sysadmin going crazy on me)? OR, is there a way to synchronize the two processes? Thanks, Gerry. P.S. - I saw this thread on the CFD-Online forum and tried the options, but it did not change the 30-second lag.

July 11, 2018, 02:40		#3
Gerry Kan Senior Member Gerry Kan Join Date: May 2016 Posts: 347 Rep Power: 10	It looks like it is definitely an NFS lag thing. What I find strange is that: a) In externalMixedFvPatchField::createLockFile ( ), a flush() is called immediately after creation of said lock file. So I don't know why it takes another 30 seconds to get picked up by the other process through the NFS. But then again, OF looks for the lock file from the other process anyways. b) On my other process (externalSolver in the tutorial) I made a call to sync after touching the lock file. Still, it took OF about 10 - 20 seconds to pick it up. So I guess the syncing worked, but not as much improvement as I wanted. In the meantime I am just going to change the BC definition and have OF skip the lock file checking. At the moment I am not worried about generating the BCs so it is all good. Gerry.