[PattiMichelle]

I have recently had some trouble with OpenSuSE 12.3x64 doing HPC on an opteron system. I'm wondering if anyone has an idea of the best distro/kernel for HPC on a big NUMA box, like the multicore Opteron systems? Basically, the issues I think are stability and load balancing.

So far I am looking at CentOS, Kubuntu, Linux Mint, and LMDE. I would appreciate anyone sharing their experience. CentOS and Linux Mint seem to run well.

Thank You,
Patti

[jdrch]

The best pros I know use CentOS, but Linux Mint is by far the most user friendly option.

[marek_ac]

I confirm - Centos is very good - stable and easy to configure. I use it to compile CFD sources written in FORTRAN, C and CUDA. DB2 is working OK as well.

[Linse]

The best kernel certainly would be the one adapted to the machine in question by someone who knows how to adapt kernel configuration.

Apart from that, I would suggest having a look at the main user clientele of the different distributions:
- (K)ubuntu and Fedora for people who want to have easily usable desktops supporting most up-to-date software
- Suse for people who want slightly more stable systems with some hardwere less supported
- RedHat Enterprise, CentOS, Scientific Linux (basically all the same) designed for server and HPC usage, considering the users to be somewhat linux-literate

From my experience, I would suggest CentOS or Scientific Linux for your use, combined with a week for getting used to the administration of the system, if you are not yet linux-literate.

[PattiMichelle]

Thanks for the information - I was getting that same opinion when I was trying out CentOS7 - it seemed very good at all hardware detection, and it (along with Mint 17) were the most stable on a flaky system I'm troubleshooting.

My second-order concern is how well the OS does Opteron 63xx NUMA since I'm doing big parallel simulations. Things like load balancing, and automatically using memory on the same socket as the CPU using it (I think all late kernels do this well). I know that CentOS mods the kernel, and so uses older kernels - Mint basically uses the latest kernel... I'm not really good enough to use numactl and OpenMPI's ability to do this effectively, since the grid is broken into pieces and placed on separate processors (not my code), but it seems to me that one would want adjacent grid pieces only one HT (hypertransport) hop away from each other to share boundary data (not all CPU's are one HT hop away from each other on quad-socket Opterons). Since I didn't write the domain decomposition software, I sort of have to rely on the OS to detect memory/CPU. Of course, this may not be terribly important, for all I know.

http://forums.debian.net/viewtopic.php?t=77247
I'm not exactly linux-illeterate, I do fstab, sftp/ssh, etc., but things like using numactl are pretty new to me. I tend to use user forums, and CentOS's user forum seems very slow. I do know that server vs. HPC also have slightly different requirements - server wants a lot of independent, stable threads and IO, but not necessarily using 100% CPU/memory constantly and cooperatively - which is more like HPC.

EDIT: As an aside, is ACML (AMD Core Math Library) a Good Thing when compiling in gfortran on such a machine? Is it difficult to do?

THANK YOU!!
Patricia

[mprinkey]

There is no way to know which kernel is going to serve your needs best. We usually start by building the latest stable version from kernel.org and then benchmark exhaustively. There are many moving parts aside from the kernel. OpenMPI, OpenIB, library, and compiler versions can all substantially effect performance. Be careful to not optimize on one variable at a time--the axes are not orthogonal. There are often complicated compatibility issues among those different components, so this is not a trivial problem to solve. Read as much as you can about the various NUMA features in kernel revisions and try to use the latest version available. Poor NUMA support is going to ruin any hope of extracting good performance from your hardware. Then move to older kernel versions only to satisfy compatibility constraints imposed by OpenIB, OpenMPI (or whatever MPI library you use), etc.

CentOS is a solid distribution. We use OpenSUSE, but that is largely for historical reasons. You will see the best 3rd party compatibility with CentOS because RHEL is generally *the* target distro that application vendors target. We primarily have Intel hardware and use the Intel compiler toolchain and math libraries, so I can't offer much insight into your AMD-specific questions.

Good luck.

[PattiMichelle]

Originally Posted by mprinkey

CentOS is a solid distribution. We use OpenSUSE, but that is largely for historical reasons. You will see the best 3rd party compatibility with CentOS because RHEL is generally *the* target distro that application vendors target. We primarily have Intel hardware and use the Intel compiler toolchain and math libraries, so I can't offer much insight into your AMD-specific questions.

Good luck.

Thanks for the discussion, Michael! I use OpenMPI (most of the FORTRAN codes I use for CFD use OpenMPI and a little bit of OpenMP). I guess I was looking for something big, rather than a lot of fine-tuning. It sounds like CentOS is probably the best bet as there appear to be a lot of performance and stability tweaks. I had seen AMD Core Math Library was recommended for CFD on Opterons when building FORTRAN OpenMPI codes. Do you have any experience with that? It appears to be as easy as adding a library link to gfortran.

EDIT: Oops, maybe not. I'm not sure about how ACML syntax is different from "standard" gfortran syntax...

[mprinkey]

Sorry. We have no AMD systems and usually do not use gcc compilers for production code. The code produced by Intel compilers is often significantly faster than gcc suite binaries. I would encourage you to look into other compilers for AMD (PGI, Pathscale, etc.). Performance improvements can be substantial and many vendors will give you temporary licenses to evaluate them.

[PattiMichelle]

Originally Posted by mprinkey

Sorry. We have no AMD systems and usually do not use gcc compilers for production code. The code produced by Intel compilers is often significantly faster than gcc suite binaries. I would encourage you to look into other compilers for AMD (PGI, Pathscale, etc.). Performance improvements can be substantial and many vendors will give you temporary licenses to evaluate them.

Thank you again, Michael - I hear that comment a lot. The Intel suite configurations seem confusing to me, especially trying to use ACML with Intel. I noted many of the supercomputers are Opteron based, I guess I could ask a Guru at some supercomputer center the same question... Just out of curiosity, do you use the Intel MPI (say, that comes with the parallel suite) or OpenMPI? Intel CC or gcc (for fortran codes)? (I don't really understand that last question, but a lot of the makefiles for FORTRAN NASA climate/weather models give mixed compilation as an option.)

I would like to try various configurations out, but I've spent days in the past trying to get the Intel suite up and running under Linux and never did get a successful build. So to try several combinations would be person-months of effort, I would guess (hence the questions)... Thank You!!!

EDIT: Intel Math Kernel Library or AMD Core Math Library?

[mprinkey]

I don't know for sure, but I believe that Intel has effectively broken the binaries its compiler suite/math kernel libraries produces when run on AMD processors. I know there was a huge dust-up about this a few years ago and the situation may have improved. But whatever the case, Intel is not in the business of making code that runs well on AMD processors. Therefore, I would avoid trying to fit an Intel square peg into an AMD round hole. As I said, there are other compilers that directly target AMD processors, including one supported by AMD itself.

http://developer.amd.com/tools-and-s...ompiler-suite/
http://developer.amd.com/partners/to...ers/compilers/

Again, I only point to these as options that I would pursue. Your priorities may be different, and your time may well be better spent in other ways. As you correctly note, compiling and benchmarking with different compilers, math libraries, and communications libraries can be both time consuming and exhausting.

Good luck.

[PattiMichelle]

Thanks, I appreciate the links!! Capitalism shows its colors... but I found a benchmark...
http://www.phoronix.com/scan.php?pag...c_open64&num=1

This is SUCH an old question, and there are many, many, many Opteron HPC systems out there. There must be a good answer somewhere. This is the Internet after all... :-/