[Simbelmynë]

Hey,


I'm playing around with my home cluster.


MX Linux 18.1. OpenFOAM v6 compiled from source.



It's a heterogeneous setup with a 7600k, 3930k and 2600k.


The motorbike benchmark on either the 7600k or the 3930k gives approximately the same results which seems reasonable (3200 MHz memory vs 1600 MHz).


However, when I run the case with both the 7600k and 3939k it becomes much slower.


If I use as much resources as possible (N=14 over the 3 computers) then I manage to get approximately 10% faster results compared to just using the 7600k or 3930k.


I use NFS for the simulation directory as well as to share the OpenFOAM installation.


Ethernet.



I have tried -bind-to-core and it makes no difference. Any other switch I should try?



looking at the output from simpleFoam it seems that the update between outer iterations is taking a lot of time. Can this be due to some I/O issue?


Any pointers would be welcome


** Update **


So with only two computers and N=10 it seems that the scaling is way better.



It finish the motorbike testcase in 139 s, which seems OK given the hardware.

[evcelica]

You need to balance the computing load with the computing speed of each node. Does openfoam allow you to do this? Or can you only assign number of cores for each computer. You should probably run benchmarks of each machine separately to figure out their relative speed, and how well they scale on different number of cores by themselves. Be aware this can change with problem size, double precision, and other factors.

This is why a homogeneous cluster is always preferred, as the loads are always matched perfectly to the relative speed since they are all the same.

[Simbelmynë]

Agreed. These are just some random parts I had lying around. The 3930k and 7600k are evenly matched in terms of performance. I think the first tests where I used N=4 (2 on each CPU) might be misleading since the 3930k probably lags behind in that scenario. It needs 6 cores to be competitive it seems. And with faster memory I think it will not be able to fully use the 4 memory channels. The 2600k is way slower on it’s own so that may be the cause for the poor performance when not using load balancing.

I will do some more testing. Anyways, the added 3930k effectively increased the computing power of my home setup to - faster than a 7940X, for virtually no money at all.

Now, since both memory and CPU of the 2600k can be pushed quite a bit, perhaps I can add some more punch to the setup if I manage load balancing.

[wkernkamp]

Originally Posted by Simbelmynë

Agreed. These are just some random parts I had lying around. The 3930k and 7600k are evenly matched in terms of performance. I think the first tests where I used N=4 (2 on each CPU) might be misleading since the 3930k probably lags behind in that scenario. It needs 6 cores to be competitive it seems. And with faster memory I think it will not be able to fully use the 4 memory channels. The 2600k is way slower on it’s own so that may be the cause for the poor performance when not using load balancing.

I will do some more testing. Anyways, the added 3930k effectively increased the computing power of my home setup to - faster than a 7940X, for virtually no money at all.

Now, since both memory and CPU of the 2600k can be pushed quite a bit, perhaps I can add some more punch to the setup if I manage load balancing.

Since you are running openfoam with the scotch partitioning, you can try weighting. See //https://www.openfoam.com/documentation/user-guide/running-applications-parallel.php


If you do try, let me know if it is effective. Haven't used it myself.