[Stabum]

Hello,
I'm trying to build a "workstation", in order to carry on my studies in Fluid Mechanics even when I'm at home.
I'm using ANSYS CFX and Fluent and the licence I can use is 8 core limited.
For this reason I've decided to go for a non Xeon processor: I'd like to get an Intel i7 5960x.
My main problem is now the memory. I'm thinking on a 64GB DDR4 memory kit, split in 8 modules. This should give me enough bandwidth (as I've read in some extremely interesting posts).

THE QUESTIONS ARE:
1) is the operating frequency really IMPORTANT???. The RAM type I'm thinking about works at 2666MHz. But there's a cheaper one which works at 2133 MHz. Can this AFFECT a 6-12hrs simulation???

2) The fact that I've chosen a non Xeon processor means that I'm going to use non ECC RAM memory. Do you think I'm committing a mistake?

Thanks so much for your advices, in computer hardware and, even more, in CFD.

Best regards

[flotus1]

The operating frequency of your RAM does matter.
In fact, the memory frequency appears as a factor when computing memory bandwidth. And as you might have read, memory bandwidth is one of the factors that affect performance for CFD calculations. Of course there are other factors like latency, but modules with higher maximum frequency can usually be operated with lower latencies aswell.

Going for non-ECC RAM seems to be the better choice in your case. ECC RAM is the only reasonable choce for servers or for workstations with a huge amount of RAM running simulations that take several weeks or are "mission critical". But the downside of ECC RAM would be a maximum frequency of 2133MHz and a non-overclockable Xeon processor. Altogether, I would estimate that with the non-ECC option you get a system that is around 30% faster (or more if your CPU sample is a good one) for the same amount of money.
Since your simulation times are rather short you probably wont encounter any memory-related errors at all or at least if such an error should occur only a few hours of simulation time are lost. Of course if losing a simulation and having to start over again is intolerable for you even once, choose ECC RAM. And you should be familiar with the basics of overclocking a CPU in order to make use of the I7-processor.

Since you already know that you want to start out with 64GB of RAM I would suggest to start with 4 modules of 16GB each. Not populating all DIMM slots is usually considered to increase the systems stability and it leaves the option to upgrade to 128GB later. The fastest 16GB modules I know of are specified for 3000 MHz and wont cost you much more than the same amount of RAM in 8GB modules.

Of course if the price is not a problem, you can build an even faster dual-Xeon workstation.

[Stabum]

Originally Posted by flotus1

The operating frequency of your RAM does matter.
In fact, the memory frequency appears as a factor when computing memory bandwidth. And as you might have read, memory bandwidth is one of the factors that affect performance for CFD calculations. Of course there are other factors like latency, but modules with higher maximum frequency can usually be operated with lower latencies aswell.

Going for non-ECC RAM seems to be the better choice in your case. ECC RAM is the only reasonable choce for servers or for workstations with a huge amount of RAM running simulations that take several weeks or are "mission critical". But the downside of ECC RAM would be a maximum frequency of 2133MHz and a non-overclockable Xeon processor. Altogether, I would estimate that with the non-ECC option you get a system that is around 30% faster (or more if your CPU sample is a good one) for the same amount of money.
Since your simulation times are rather short you probably wont encounter any memory-related errors at all or at least if such an error should occur only a few hours of simulation time are lost. Of course if losing a simulation and having to start over again is intolerable for you even once, choose ECC RAM. And you should be familiar with the basics of overclocking a CPU in order to make use of the I7-processor.

Since you already know that you want to start out with 64GB of RAM I would suggest to start with 4 modules of 16GB each. Not populating all DIMM slots is usually considered to increase the systems stability and it leaves the option to upgrade to 128GB later. The fastest 16GB modules I know of are specified for 3000 MHz and wont cost you much more than the same amount of RAM in 8GB modules.

Of course if the price is not a problem, you can build an even faster dual-Xeon workstation.

Thanks SO much for your reply. This gives a lot of light to the darkness I was lost in.
I understand that not populating all DIMM slots can increase system stability: it's something I've already read when it comes to overclocking strategies and advices. But I've also read in many posts here that one of the most common mistakes in building a dual cpu workstation is using a small number of memory sticks for a given amount of RAM, because this should produce a bottleneck. Moreover I've searched a lot in a famous online shop (I think I cannot indicate it here) and there's NO 16GB sticks sold there (only ECC 16GB Sticks I mean).

I've checked out in the motherboard manual (Asus X99 PRO) and I found that the G.Skill ram type I wanted is in the QVL list. Do you think I should be safe with them?
Which is, in detail, the RAM module you were talking about in your previous post?

Thank so so much again!

[flotus1]

The issue with "not enough DIMMs" is only relevant when the number of DIMMs is lower than the amount of memory channels. For your quad-channel I7 processor, 4 DIMMs are enough to use the maximum bandwidth. More than 4 DIMMs will not increase performance.
This is obviously different for dual- or quad-socket systems. To use the full bandwidth of 2 CPUs each with a quad-channel memory controller, you need at least 8 DIMMs and you have to populate them correctly. And so on...

I guess it is ok to link some products here. Here is a selection of 16GB modules I could buy.
http://geizhals.eu/?cat=ramddr3&xf=5...54_2666&sort=r
It would surprise me if there is really no online shop in your part of the world that sells any of these. Edit: I see you are in Italy. Some of these shops surely ship to Italy aswell.
Yet using RAM the motherboard manufacturer tested and approved for compatibility is the easiest option.

[Stabum]

Originally Posted by flotus1

The issue with "not enough DIMMs" is only relevant when the number of DIMMs is lower than the amount of memory channels. For your quad-channel I7 processor, 4 DIMMs are enough to use the maximum bandwidth. More than 4 DIMMs will not increase performance.
This is obviously different for dual- or quad-socket systems. To use the full bandwidth of 2 CPUs each with a quad-channel memory controller, you need at least 8 DIMMs and you have to populate them correctly. And so on...

I guess it is ok to link some products here. Here is a selection of 16GB modules I could buy.
http://geizhals.eu/?cat=ramddr3&xf=5...54_2666&sort=r
It would surprise me if there is really no online shop in your part of the world that sells any of these. Edit: I see you are in Italy. Some of these shops surely ship to Italy aswell.
Yet using RAM the motherboard manufacturer tested and approved for compatibility is the easiest option.

Grazie Grazie Grazie !!!!

I checked out in Amazon.it twice and, believe, there's no 16GB NO-ECC RAM Modules available. Moreover, in the Motherboard Owner manual you can read: "8xDIMM DDR4 max at 64GB", which could mean:
1) the biggest memory modules you can use are 8GB modules;
2) the 16GB modules were not on the market at the moment of publishing.

If you were in my shoes what would you do?


Really many many thanks for all the time you have spent with me today!

[flotus1]

If I were you and had some time to kill I would ask Corsair which motherboard they recommend for their 16GB modules and if 4 and 8 of them together will work.
What you should do anyway is check if quad-channel memory is supported with 8 of the modules you picked. Some manufacturers only guarantee quad-channel mode with 4 DIMMs, not with 8 for some memory types.
http://www.asrock.com/mb/Intel/X99%2...asp?cat=Memory

After all, 128GB of RAM are reported to work but are not even officially supported by the Intel I7 processors. So if you know that you wont need more than 64GB of RAM you might just skip the hassle and get something from the manufacturers compatibility list.

Just a comment on the whole project: If this is the first system you build on your own and you have no expert around you can ask, maybe you are better off buying something off the shelf or from a pc supplier near you. Otherwise some of the problems that could occur might just be beyond your troubleshooting abilities.

[Micael]

Hello,

i7-5960X is a good choice for running FLUENT with a limited 8 hpc licence and a moderate budget.

128GB RAM does not make much sense though for such a system. If one needs that much amount of memory, then he/she needs much more than 8-core (or being ready to wait ages for results).

[Stabum]

Originally Posted by flotus1

If I were you and had some time to kill I would ask Corsair which motherboard they recommend for their 16GB modules and if 4 and 8 of them together will work.
What you should do anyway is check if quad-channel memory is supported with 8 of the modules you picked. Some manufacturers only guarantee quad-channel mode with 4 DIMMs, not with 8 for some memory types.
http://www.asrock.com/mb/Intel/X99%2...asp?cat=Memory

After all, 128GB of RAM are reported to work but are not even officially supported by the Intel I7 processors. So if you know that you wont need more than 64GB of RAM you might just skip the hassle and get something from the manufacturers compatibility list.

Just a comment on the whole project: If this is the first system you build on your own and you have no expert around you can ask, maybe you are better off buying something off the shelf or from a pc supplier near you. Otherwise some of the problems that could occur might just be beyond your troubleshooting abilities.

I'm going to contact the Asus customer service and ask wether they advice using all the 8 dimms available, and if they advice using 16 GB ram modules. This will not be my first build at all. I've been building my rigs for 10 years now and, probably because I've always choosen good quality components, everything had always worked flawlessly. For sure it's going to be the most expensive one, but I just hope that putting all my care, everything will work fine. And a part of the success, for sure, will belong to you.
Again, thanks so much!

[Stabum]

Originally Posted by Micael

Hello,

i7-5960X is a good choice for running FLUENT with a limited 8 hpc licence and a moderate budget.

128GB RAM does not make much sense though for such a system. If one needs that much amount of memory, then he/she needs much more than 8-core (or being ready to wait ages for results).

Thanks so much for your comment!
So far I've always used 32gb ram machines. 64gb is a huge upgrade and, frankly, I just can't imagine myself using 128 gb... Also from this aspect I should measure the distance between a student like me and CFD professionals like you guys.

[Stabum]

Originally Posted by flotus1

If I were you and had some time to kill I would ask Corsair which motherboard they recommend for their 16GB modules and if 4 and 8 of them together will work.
What you should do anyway is check if quad-channel memory is supported with 8 of the modules you picked. Some manufacturers only guarantee quad-channel mode with 4 DIMMs, not with 8 for some memory types.
http://www.asrock.com/mb/Intel/X99%2...asp?cat=Memory

After all, 128GB of RAM are reported to work but are not even officially supported by the Intel I7 processors. So if you know that you wont need more than 64GB of RAM you might just skip the hassle and get something from the manufacturers compatibility list.

Just a comment on the whole project: If this is the first system you build on your own and you have no expert around you can ask, maybe you are better off buying something off the shelf or from a pc supplier near you. Otherwise some of the problems that could occur might just be beyond your troubleshooting abilities.

Dear Flotus,
I've chatted with a guy of the Asus customer services today and now everything seems clear.
The reason why it's so hard to find a 16GB RAM stick in Amazon.it is that I was searching in the wrong place: Haswell Extreme Motherboards do not officially support these memories. Z170 Skylake Motherboards do support them. You'll never find, in any X99 Mobo Owner manual that a 16GB memory stick is officially supported, no matter how many sticks you're going to use, no matter the frequency you're going to use them at. Now, the fact that these memories can be used as well is a completely different story, but if you're looking for reliability (as I am!), you will never risk to adopt a memory module which has never been tested with your motherboard before. In case of troubles you're on your own...

Morevoer, if you look at the Asus X99-Sabertooth (which is my definitive choice) manual, you'll see that there's a particular type of Corsair Vengeance LPX which has been tested at 2400MHZ with all the 8 DIMMS busy (64GB pack). For sure I'll definitely go for it!!!

So, at the end of the day, 8 dimms busy, no room for future upgrades (I doubt I'll ever need them), 64GB RAM@2400MHZC14 officially tested by Asus.

Thanks so much for your precious help and advices!!!

[flotus1]

I see. Thanks for sharing the information.
I guess you are referring to this QVL for the mainboard you picked. It lists a few even faster memory types that are approved for using 8 DIMMs.
http://dlcdnet.asus.com/pub/ASUS/mb/LGA2011/SABERTOOTH-X99/SABERTOOTH_X99_DRAM_QVL_20150108.pdf
Since CFD is one of the few applications where faster memory actually affects performance and thus actually makes sense you might aswell get one of the faster kits. 3000 MHz compared to 2400 MHz is a 25% increase in bandwidth assuming similar latencies and will make the system only slightly more expensive if at all.

[jmorales]

Hi I have a few recommendations...

1) if you have a budget go for a 5960X intel core i7 or dual Xeon e5 2660 V2
and get a Supermicro X9dai and use DDR3 RAM ECC, is very cheap.
2) ECC RAM is not necessary, at least no 100%, this type of memory just prevents some error of data allocation, if you are not running highly important data, you could pass of ECC ram.
3) 2400 Mhz RAM i think is ok, and for now is not so expensive to get 64gb on amazon or eBay.
4) if you are running CFX or fluent, go for a GPU with high level of GPU computing, to perform faster calculations, a GTX Titan, Titan Black are the best in this field, with 4,7 Tflops Single Precision and 1,5 Tflops Double Precision, this can help a lot to improve the total time calculation, if you know how to configure well the software

[Stabum]

Thanks for all your recommendations, which will be extremely useful. One question: you advice to go for a GTX TITAN, in order to exploit its gigantic computational power... BUT I DON'T KNOW HOW TO USE THE GPU for calculations!!! Can you help me, pleeeeeese? 😊😊😊

[jmorales]

you have several options...

1) if you are under Fluent Solver, just open Fluent and select Parallel Computing, select the numbers of GPU that you have and that is all, but, this method only uses 15-20% of the GPU computing performance, but is the easiest way.

2) If you are under CFX you need to configure the solver to use GPU but honestly i don´t remember the setting exactly, and if you have knowledge of Code Programming you can edit some features in the code to call the GPU for the calculatiion.

3) If you are using fortran or other codes platform for fluid dynamics is 100% code commands to call the gpu.

4)... let me install again Fluent in my new computer and perform some tests and maybe i will answer you in a more formal way

Check this link also....

http://www.nvidia.com/content/tesla/...-userguide.pdf

[HyperNova]

hi,
i7 5960x is a powerful cpu for cfd simulation, by overclocking it to 4.3 at least its power enhances up to 30 percent of its standards, here is a pic of it

good luck and enjoy your system

[Stabum]

Originally Posted by HyperNova

hi,
i7 5960x is a powerful cpu for cfd simulation, by overclocking it to 4.3 at least its power enhances up to 30 percent of its standards, here is a pic of it

good luck and enjoy your system

Hello Hypernova!
I've finished building my PC. Here are the components:
Motherboard: Asus X99 Sabertooth
CPU: Intel I7 5960x
RAM: Corsair Vengeance DDR4 CL9 2400MHz 64GB
CPU Cooler: Noctua NH-D15
PSU: Corsair H850M
GPU Nvidia Quadro K4200

I like the idea to overclock it. But i'm not an overclocker.
Could you pass me the parameters of your overclock?
It would be SOOO useful for me...
Thanks so much!

[HyperNova]

Hi Stabum,
first i hope it will work fine for you
we bought our system from a company and they overclocked it (we do not know how they did , i have heard they do something with cpu multiplier), the company claimed the performance increase about 30% which is a lot and they said it is stronger than 2*xeon 2630v3.
In my opinion for your system it would be a bit better if you had bought 3000Mhz ram which is faster.
Good luck friend

[flotus1]

Using overclocking settings from different machines will not necessarily work for you.
If you are not familiar with the basics, this video might contain everything you need to know for basic overclocking.
https://www.youtube.com/watch?v=Nj4ixZE7EWo

You can guess from the style of presentation that this video is not intended for professionals, so some of the conclusions do not apply to you. Especially the part about RAM frequency and timings, which do really matter in CFD.
Using the search term "Haswell-e overclocking" you will find more detailed instructions for your endeavor. I highly recommend reading some of these guides before you start if this is the first time you try something like this.

[leasken]

Originally Posted by Stabum

Hello Hypernova!
I've finished building my PC. Here are the components:
Motherboard: Asus X99 Sabertooth
CPU: Intel I7 5960x
RAM: Corsair Vengeance DDR4 CL9 2400MHz 64GB
CPU Cooler: Noctua NH-D15
PSU: Corsair H850M
GPU Nvidia Quadro K4200

I like the idea to overclock it. But i'm not an overclocker.
Could you pass me the parameters of your overclock?
It would be SOOO useful for me...
Thanks so much!

Nice! May I ask, how much was the total cost of this build?

[Stabum]

Around 4000 euros. There are 4 Caviar Black HD in RAID O, a 500 GB Samsung 850 Pro SSD and a couple other things...