CFD Online Logo CFD Online URL
Home > Forums > Hardware

Would you suggest Infiniband?

Register Blogs Members List Search Today's Posts Mark Forums Read

LinkBack Thread Tools Display Modes
Old   September 18, 2016, 10:07
Default Would you suggest Infiniband?
Join Date: Aug 2009
Posts: 62
Rep Power: 9
Jonny6001 is on a distinguished road

We currently have a Cisco cluster which is made up of 4 nodes. Each node has 2x Intel E5-2630v3 CPU which are 8 cores each@2.4GHz. There's 64GB or RAM in each node (8x8GB). The nodes are in a Cisco rack which I believe has a 2x10GB/s connection between the nodes.
One of the nodes is also used as the headnode.

I am looking to upgrade the cluster and have around 20,000 to spend which I'd like to get the best performance for the money.

The cluster is used mostly for a mix of Star-CCM+ and Converge CFD in-cylinder simulations with detailed chemistry solver.

I was thinking was of purchasing 4 more nodes with 2x E5-2630 v4 CPU (10 cores @ 2.4GHz) in each node, giving an additional 80 cores. Along with 8x8GB of RAM per node. I believe that the additional nodes can be installed in the same Cisco rack and use the 2x10GB/s connection.

Do you think that my plan is a sensible way of proceeding? Would we benefit from spending some of the money on Infiniband connection between the nodes?
We are limited to purchasing Cisco hardware.

Thanks for your time.
Jonny6001 is offline   Reply With Quote

Old   November 3, 2016, 08:20
Default CFD hardware
New Member
Join Date: Jan 2012
Location: USA
Posts: 9
Rep Power: 7
boryan1965 is on a distinguished road

For the price range you have in mind and tech data you have mentioned,

I would recommend to get in touch with Totalsim, are based n Brackley.

My company have bought a similar cluster configuration from them,which has worked two years without a single glitch.

Best regards,
boryan1965 is offline   Reply With Quote

Old   November 3, 2016, 13:37
New Member
Join Date: Dec 2010
Posts: 4
Rep Power: 8
monkeymax is on a distinguished road
You always want to remove bottlenecks where they exist and in my experience, network latency and bandwidth is a common bottleneck.
If you want to test just how much, run a job in STAR-CCM+ . Run that job on 16 cores, specifically filling one node completely. I suggest a single-phase simulation with relatively simple physics as these scale well and if you keep it relatively light - say 2 million cells - you're not pushing the RAM too hard either.

Then rerun that job again on 16 cores, but this time 8 cores on one node, and 8 cores on another. If you can, fully reserve both nodes so no other jobs run on it. (You could submit a job with 32 cores - i.e. both nodes full - but specify a machine file for STAR-CCM+ so it only runs on 8 cores on each node)

Time each run. STAR-CCM+ includes reports now that allow you to report the simulation or iteration elapsed time (but ideally you'd write a simple Java macro that:
- gets system time
- runs simulation
- gets system time
- computes run time from (end-start) time.
This latter method again reduces overhead and prospect of additional bandwidth being used across the network.

Do this test a few times in each configuration.

As the only difference is running on network versus running 'locally'* this should give you an idea of how much of a bottleneck the network is. I'd be surprised if it's none at all. Bear in mind more complex cases with combustion where there will be lots of inter-CPU communication will exacerbate this bottleneck significantly.

In my opinion, if you can get Infiniband, then do so. But as with most things, this is always going to be case/system dependent!

*using the consideration that two CPUs in one node is equivalent to one local run - that's not really the case, but I'm trying to keep this test simple!
monkeymax is offline   Reply With Quote


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Infiniband How-to for OpenSuse? derekm Hardware 9 April 2, 2017 10:25
Solver for FSI strongly coupled charbel OpenFOAM Programming & Development 58 December 7, 2016 22:31
how to confirm that I have already use infiniband in OpenFOAM? Detian Liu OpenFOAM 3 September 5, 2013 08:34
Fluent 12 on Infiniband network kami146 FLUENT 0 December 7, 2011 06:51
Interesting behavior when using LAM without ib rpi on an infiniband cluster OF 141 msrinath80 OpenFOAM Running, Solving & CFD 0 October 2, 2007 16:31

All times are GMT -4. The time now is 03:23.