Benchmark fpmem
 

The STREAM benchmark test the memory bandwidth, even though floating point operations are made. In the benchmark, the number of floating point operations doesn’t exceed the number of loads. This is likely also the case for many CFD programs, but for higher order solvers based on Cartesian grid, it is not the case. The ratio between number of floating point operations and loads, could be much larger for such solvers.
Optimizing in HPC is often about minimizing the reading from memory. The work can be split into smaller chunks, where as much work as possible is done on each chunk, before the next chunk of memory is processed. The relevant size of such chunks should be determined.

I have made a benchmark, fpmem, that gives the floating point performance for various combinations of floating point operations pr load, and the size of the array processed. The benchmark doesn’t do any real work, but it can be compiled, linked and run in about 5 minutes.

The instructions for compiling, linking and usage of the benchmark is given in the first few lines of the source file. It requires a resent C++ compiler (-std=c++17) and mpi. It uses AVX2 when compiled with -D_USE_INTRINSIC. See instructions.

I hope that some care to use the benchmark and post the results. The benchmark is made to run on one CPU, and if used on a large cluster the performance will just increase linearly with the number of CPUs. I don’t have access to EPYC Milan or newer Xeons on socket LGA4189 so for me results from these could be very interesting.

I have attached the benchmark (fpmem.c) and the results for my newly build system with an Intel i5-12600.


Edit: I have uploaded a new version, that corrects an error that effected the reported performance values with up to about 10%.

Here are some results from my system (2x Epyc 7551). First one with all 64 threads, second with only 8 threads pinned to the first CCD. The latter makes it behave like a single 1st-gen Ryzen CPU with very low clock speeds and extremely crappy memory.
Attachment 88587
Attachment 88588
I will leave interpretation up to you :confused:

Compared to the i5-12600 it looks like the two systems have about the same ratio between compuational performance (with AVX2) and memory bandwidth. The zen-core has one fmadd 'engine' (8 FLOPs/cycle) whereas zen2, zen3 and newer Intel cores have two 'engines' and do 16 FLOPs/cycle with AVX2. It is seen especially for the small arrays that can be contained within the first level cache. The zen cores are about half performance of the Intel cores at the same clock speed, but the 7551 has a lot of cores, Looking at the line for 64MB arrays, it is seen that both systems are memory bound up to about the point where there are 32 FLOPs/load. It is seen since the performance numbers doubles each time the FLOPs/load ratio doubles. The data supplied by the memory is the limiting factor. At 64 FLOPs/load both systems are cpu-bound. Thanks for runing the benchmark!

Another benchmark? Count me in! ;)

Not sure what it means but here you go:

I can understand there is a need for an explanation on how to interpret the results. I took it in steps. First I ran STREAM with different array sizes to test the cache and memory bandwidths. Then I looked at floating point performance in cases with several floating point operations for each load. In HPC, some problems have a low value for the ratio FLOPs/load, and others a very high value. For CFD the ratio is usually at a low value, but for matrix multiplications between two large matrices the ratio has a very high value. For small values the performance are limited by the memory bandwidth, and for high values the performance is limited by the cpu’s ability to crunch numbers. My interest is typically in the intermediate range, say with ratios from 4 to 64, where it is not evident what limits the performance.

I don't know how to include a text file, but I have attached the results for the i5-12600 again. Please look at it. Looking at the first column for a ratio of 0.5, it is seen that the performance is highest for very small arrays. Arrays at 8kB and 16kB can be contained within the 1st level cache, and it is the fastest cache. At 32kB the performance is lower, since the the 1st level cache is a little too small and the bandwidth starts to be limited by the slower 2nd level cache. From 64kB to 256kB the performance is nearly constant and determined by the bandwidth of the 2nd level cache. For larger arrays the bandwidth of the 3rd level cache starts to play a role, but from array sizes of 8MB and larger, the performance is limited by the bandwidth of the RAM. All performance figures in the first column is in this way determined by the bandwidth to the memory system in which the arrays can be contained. The calculation include two equal sized arrays, but the size specified in the table is for each array.


The column at the right for the ratio at 64 is much easier to interpret. Here all performance figures are about the same, independent of the array size. The performance here is alone determined by the cpu’s ability to crunch numbers.


For most intermediate columns the performance is about constant for the smaller array sizes, where the cpu is the limiting factor, but at some point, the memory system that contains the larger arrays gets too slow, and then the performance shifts to be limited by the memory bandwidth. The benchmark shows where this happens!

You can wrap code tags around any text you want to appear as formatted in a plain text file.
[CODE] text goes here[/CODE ] <- remove the space
If I understood correctly, this program is partially a memory/cache benchmark. Pretty much the triad part of stream, without possibility for streaming stores. Might be neat to have another output for bandwidth.

I have made a version that also shows the corresponding memory bandwidth. There were a lot of figures before and even more now.....


The figures show my point with the benchmark. Looking at the line for the bandwidth results for an array size of 8MB, then the bandwidth is almost constant for the first 7 results, and then drops for the last result. It indicates that the fp performance is memory bound up till the result for 64 FLOPs/load, where it is the number crunching that is the limiting factor. For the 1MB array, the bandwidth is at a constant level for the first 4 results, and then it decays. In other words, the computational performance is limited by memory bandwidth for the first 4 results, and then for higher ratios of FLOPs/load it becomes computational bound.

Looking at the column for 0.5 FLOPs/load, then the performance of the memory system can be seen. From 64kB to 256kB the results are almost constant, showing the performance of the second level cache. From 8MB and up, it is the bandwidth of the RAM that limits the performance. In the results for the bandwidth I have include the one write for each two reads like in the STREAM benchmark. The performance for large arrays are very similar to the bandwidth figures from STREAM.

Looking at the computational performance, the performance has dropped a bit compared to the results previously posted. I made a mistake that effects from 4 FLOPs/load and up. The performance is reduced by about 10% for for 4 FLOPs/load, 5% for 8 FLOPs/load and 2.5% for 16 FLOPs/load. I have uploaded a corrected version in the first post. It is also corrected in the version attached here, that also reports memory bandwidth.

Recompiled with openmp and two threads per process (running 32 processes instead of 64). The GFlops are much improved because this cpu shares cache and fpu between two integer cores.

I dont understand why the GFlops dont keep increasing as the flop_load_ratio goes up. Anyone have an answer?:confused:

The Gflops do increase for your Opteron system. See the computational performance in the tabel at the buttom. The tabel at the top shows the amount of data read from memory, corresponding to the computational performance in the lower tabel. The GB/s decays when the computational performance becomes the limitting factor. That is the case, when the flop_load_ratio is high.

The performance is either limited by memory bandwidth or by computational performance. The test gives a picture of what is the limitting factor for various array sizes and flop_load_ratios. Hope this helped.

It does on my result, but not for the others. If I run the 128 case, it also drops. Why would a higher number of repeats lead to reduced flops?