[Duke711]

Is there an interesting alternative to Amazon AWS. Should there be an option to rent the computing power per hour and, above all, should setting up the individual software and OS only cost a fixed rate?


How do you calculate larger models?



Among larger models everything that would take longer than a week on a dual socket plattform.

[flotus1]

I am using this: https://www.hlrs.de/home/

Or similar institutions affiliated with universities. One of the benefits is that they are well-suited for HPC applications, and a lot of the software around CFD you might need is already set up.

[Duke711]

Originally Posted by flotus1

I am using this: https://www.hlrs.de/home/

academic or industrial usage?


pricing?

[flotus1]

Non-academic for me. They have price lists publicly available on their website. "Hawk" is probably the type of node you would want to use for CFD, the nodes are 2x64 core Epyc Rome CPUs with 256GB or memory.
https://www.hlrs.de/solutions-servic...-requirements/

[trampoCFD]

Hi Duke, what software do you use? I run a supercomputing service for STAR-CCM+. Upload/download is automated, no script is needed. The models run through a batch job manager but we have a PRE/POST processing facility in Beta if required. You can run 20,000 cores+ if your models scale.

[Duke711]

Originally Posted by trampoCFD

Hi Duke, what software do you use? I run a supercomputing service for STAR-CCM+. Upload/download is automated, no script is needed. The models run through a batch job manager but we have a PRE/POST processing facility in Beta if required. You can run 20,000 cores+ if your models scale.

Fluent, is your service only for CCM?

[trampoCFD]

Sorry, we only do Star-CMM+ :-(

[schmichi]

Originally Posted by flotus1

Non-academic for me. They have price lists publicly available on their website. "Hawk" is probably the type of node you would want to use for CFD, the nodes are 2x64 core Epyc Rome CPUs with 256GB or memory.
https://www.hlrs.de/solutions-servic...-requirements/

Hi flotus1,


currently I'm looking for options to run some private OpenFOAM projects on.
These are

1) Building an own CFD workstation is rather unrealistic for the short term.
Did some research on hardware based on your sticky thread here in the forum and no currently available option seems reasonable for all my needs (not only CFD, if I build a new computer it should also be able to run games at a proper performance from time to time)

2) Using cloud resources.
There are several options available and HLRS at a first glance seems interesting. Actually I know the facility, but did not know that also private persons/small businesses can get on this one.
May I ask a few questions on your experience with HLRS:
- How is your overall experience as a (I assume) 'small' user?
- Aren't the available nodes on Hawk an overkill? 2*64 cores Epyc CPUs, as you will end up with 8 cores per memory channel.
According to the price list you pay in node hours (allocated node * allocated time) and not CPU hours, hence the price to performance ratio is less efficient as it could be.

Even though I'm a somewhat experienced CFD user, I'm rather new to the hardware side for CFD. So I might very well have missed something.

Thanks in advance for your feedback.

Cheers

[flotus1]

1) Building an own CFD workstation is rather unrealistic for the short term.
Did some research on hardware based on your sticky thread here in the forum and no currently available option seems reasonable for all my needs (not only CFD, if I build a new computer it should also be able to run games at a proper performance from time to time)

A "one size fits all" solution for your requirements is definitely tricky. But not impossible, depending your budget, skills (i.e. building the thing yourself) and more defined requirements. You can game on server-type CPUs, just not with ridiculously high frame rates while CPU-limited. And you can run CFD on desktop CPUs, but with limits to absolute performance and memory size. It's a tradeoff.
I would accept the challenge if you want to flesh out your requirements a bit in a separate thread.

- How is your overall experience as a (I assume) 'small' user?

Definitely a small or rather infrequent user. It's been fine so far. The documentation is decent, and staff responds quickly to any questions/problems we had. They had a lot of downtime lately, which didn't affect me fortunately. Something you can probably avoid by choosing one of the hyperscalers like AWS instead.
But apart from that, I felt right at home. But be aware that you need to set aside some time in order to familiarize yourself with general operation, and getting everything set up. Especially if this is your first time using a cluster.

- Aren't the available nodes on Hawk an overkill? 2*64 cores Epyc CPUs, as you will end up with 8 cores per memory channel.
According to the price list you pay in node hours (allocated node * allocated time) and not CPU hours, hence the price to performance ratio is less efficient as it could be.

We need to differentiate between buying the hardware yourself, and renting it.
For a workstation you buy, the high core count CPUs aren't the most cost-efficient, even for free software like OpenFOAM. But when buying in bulk, and running them in a datacenter, list pricees for hardware don't matter to the same extent. I.e. they could not make the node-hours much cheaper if they had used 32-core variants of these CPUs.
Also, you still get a small benefit thanks to the larger L3 caches of the 64-core CPUs, even if scaling stops after 32-48 cores per CPU.

But let's not get lost too much in the details, because you raise a valid point:
It can be cheaper overall to run your simulations on older and cheaper nodes. You will have to do a quick benchmark yourself, and factor in the cost for each node type. I didn't bother looking into it myself, because the cost for rented computing resources is a relatively small item on our invoice.

[schmichi]

Originally Posted by flotus1

A "one size fits all" solution for your requirements is definitely tricky. But not impossible, depending your budget, skills (i.e. building the thing yourself) and more defined requirements. You can game on server-type CPUs, just not with ridiculously high frame rates while CPU-limited. And you can run CFD on desktop CPUs, but with limits to absolute performance and memory size. It's a tradeoff.
I would accept the challenge if you want to flesh out your requirements a bit in a separate thread.

It's indeed a bit tricky. Some options come to mind though and I might come back to you in a different thread as you kindly suggested.


But overall this is also, why I'm considering cloud usage. In combination with a proper but not over the top desktop workstation.

Originally Posted by flotus1

Definitely a small or rather infrequent user. It's been fine so far. The documentation is decent, and staff responds quickly to any questions/problems we had. They had a lot of downtime lately, which didn't affect me fortunately. Something you can probably avoid by choosing one of the hyperscalers like AWS instead.
But apart from that, I felt right at home. But be aware that you need to set aside some time in order to familiarize yourself with general operation, and getting everything set up. Especially if this is your first time using a cluster.

Fortunately it's not the first time for me using a cluster. I'm using one on a daily basis in my job. It's all properly set up there though, but general operation and getting everything set up should work rather quickly.
Regarding usage rate, I'd definetly see myself also on the infrequent side. Feeling 'right at home' is most probably a plus for such kind of smaller/infrequent use cases compared to the hyperscalers.

But of course also price and other topics like accessability matters. E.g. I tried to find some pricing for companies like AWS, Google Cloud, etc. to compare also with HLRS and maybe other options, but unfortunately I quickly got lost.

Originally Posted by flotus1

We need to differentiate between buying the hardware yourself, and renting it.
For a workstation you buy, the high core count CPUs aren't the most cost-efficient, even for free software like OpenFOAM. But when buying in bulk, and running them in a datacenter, list pricees for hardware don't matter to the same extent. I.e. they could not make the node-hours much cheaper if they had used 32-core variants of these CPUs.
Also, you still get a small benefit thanks to the larger L3 caches of the 64-core CPUs, even if scaling stops after 32-48 cores per CPU.

But let's not get lost too much in the details, because you raise a valid point:
It can be cheaper overall to run your simulations on older and cheaper nodes. You will have to do a quick benchmark yourself, and factor in the cost for each node type. I didn't bother looking into it myself, because the cost for rented computing resources is a relatively small item on our invoice.

Thanks for your detailed explanation here, makes sense.
In case of HLRS there are almost no cheaper options available then the HAWK, despite some Haswell nodes.
But it could make sense to check, when opting for other options like AWS.

[Gompute]

For Fluent, and any other CFD you can try Gompute. Some advantages:
- ANSYS Preferred Cloud hosting partner
- Bare metal nodes, for CAE applications only with remote visualization included.
- Fluent (and many other apps) are pre-integrated, and tested.
- There are different business models to fit your needs
- Free test

Just drop us an email at info@gompute.com and we will guide you through the testing process, or check our site: https://www.gompute.com/project/ansys/

Regards