[msrinath80]

Hi all,

I am trying to build OpenFOAM on the BlueGene. Unfortunately I hear that the BlueGene/L does not support shared libraries. My question is whether I can build OpenFOAM using static libraries (i.e. lib instead of libso). Has anyone implemented this successfully even on a GNU/Linux system?

[mwild]

Hi Srinath

That's going to be a pretty tough one if that really is the case... Problem is that OpenFOAM pretty much relies on dynamically loading shared libraries (e.g. the grid motion solvers, the new forces library and the GAMG linear solvers are examples). To get around that, you would need to do some serious surgery and make sure that you link all required libraries at link-time and not at run-time as is the case now. Further you need to fix the build system to statically link the correct libPstream.a library (currently this is done by setting LD_LIBRARY_PATH, which you can't use for obvious reasons).

Here's what I would try to do:

- Fix up the wmake build system (i.e. replace the "libso" calls in the Allwmake scripts by "lib" (I hope I got the names right...)

- Make sure that the correct Pstream library gets compiled and linked. For safety, disable the compiling of the other implementations!

- In src/OpenFOAM/dlLibraryTable/ remove all the #include's for <dlfcn.h> and replace the definition of "bool Foam::dlLibraryTable::open(const fileName& functionLibName)" by a dummy implementation which fails with a FatalErrorIn and a very distinctive message, such that you know what is going on. Also replace "Foam::dlLibraryTable::~dlLibraryTable()" such that it doesn't call "dlclose" anymore.

- Compile the thing on a "normal" computer and try to link the code statically. You will probably get some undefined references along the way, which you have to fix by adding the appropriate libraries to the relevant Make/options files.

- Once you managed to compile, you will need to make sure that the programs you're going to need on BlueGene are running. For this, just start them up. If they fail with unresolved symbol errors, go back to the last step by adding required libraries, and re-linking.

- When you run the programs and see that FatalErrorIn you inserted in the dlLibraryTable class, figure out where the code is that called that "open" function and fix it. Also add the required library to the list of link-libraries.


I hope this helps.

Michael

[msrinath80]

Thanks Michael. That sounds really complicated though. Life would have been a lot simpler if IBM used a GNU/Linux based kernel on the BlueGene compute nodes.

I'll keep you posted on any developments.

[shivasub]

Hi Srinath,

What OS is the BlueGene/L machine running?

A while ago I had compiled OpenFOAM on an IBM AIX machine. I compiled both static and dynamic libraries there.

For the static version I compiled each library by hand using wmake lib, as Michael suggests, Subsequently compiling only apps and utils which I needed.

The main issue I had with IBM's was that I had to use the native linker rather than the GNU linker.

you have to pass additional linker flags.

LINKLIBSO = $(CC) $(c++FLAGS) -shared -Wl,-G -Wl,-bbigtoc -Wl,-brtl -Wl,-bnoentry

in wmake/rules/$ARCH .

This is on the AIX, if the BlueGene/L is running AIX

Hope this helps.

-Shiva

[msrinath80]

Hi Shiva,

Thanks for your insights. I believe that the BlueGene/L head node runs Suse Enterprise Linux 9.x, while the actual compute (C) nodes run a stripped down version of AIX. We therefore cross compile the code on the head node using blrts_xl* compilers which are variants of the xl* compilers that are specifically built for the BlueGene.

[shivasub]

Hi Srinath,

Have you had any success with the XL compiler?

My solution was to use gcc with native linker. My attempt at compiling with XL threw up so many errors that it was not appetizing to go down that road.


-Shiva

[msrinath80]

Not really. I plan to use the GNU ones that were ported to the BlueGene by IBM. However, since the GNU compiler versions are 3.x at best, I am already anticipating issues with the build!

[shivasub]

I had a similar issue where the system gcc was 3.x. Compiled a 4.2 (I think or maybe 4.1.2) using 3.x and used the personal install.

You will need a custom wmake/rules for all the additional linker flags you need to pass to compile foam with native linker. I have attached the version used. I think I have the IBM manual somewhere, will dig that up and try to send that.

This is the rules tar file

[shivasub]

okay sorry about the attachment, its larger than the file size limit. Let me know if you need it, will email it to you.

regards
Shiva

[msrinath80]

Looks like I am going to need this. I'd appreciate if you could email it to me. My email address is in my profile. Thanks.

[msrinath80]

Hello there,

Thanks for all your help. I have given up on the BlueGene/L for now. This (cross-compiling with static libs) is the longest and most confusing maze I have ever wandered through. If anyone does manage to get OpenFOAM working on the BlueGene/L, please do post the instructions here.

Thanks,
Srinath

[hjasak]

I think I will get my hands onto this problem soon - will keep you posted.

Hrv

[shivasub]

Hello Srinath

Did you manage to compile 4.1.2 or higher Gcc for the AIX on the bluegene? Is it absolutely necessary to use static libs there?

As I mentioned earlier, I had compiled OpenFOAM on IBM SP4 machine running AIX. I do not have access to a Bluegene to give it a shot.

Best
-Shiva

[msrinath80]

Hi Shiva and Hrv,

Thanks for your responses. No, I was unable to build gcc 4.x for the BlueGene as it required me to port some IBM patches for gcc 3.x to 4.x which isn't trivial (for me). As a result, I downgraded to OpenFOAM 1.1 as it was built at that time using gcc 3.x. However, I was unable to progress further because of lots of compiler issues afterward.

I will await Hrv's instructions. I will have access to this BlueGene most likely only till July.

[matteoL]

Hello,
I am thinkng to compile Openfoam on Bluegene/P as well, anyone has had any luck so far?
Any sugegstions where I should start?
Are you sure that bluegene doesn't support dinamically linked libraries?It seemed to me it could..(altough I must admit I haven't really looked into it..)

Thanks,
best regards,
matteo

[ywang]

I am also working on the installation of OF on BlueGene/P......I am very confused.....