Thank you a lot, Alberto. In OpenFOAM, BICCG is diagonal ILU preconditioned BiCG solver, so it can be used to solve an asymmetrical or symmetrical matrix. :)

FYI, some updates.

1. I failed to use PGI compiler to recompile the OpenFOAM, lots of errors. And ICC, I failed again, something strange wrong.

My system info: 64 bits AMD Opteron(tm) 8347HE 1900MHz, InfiniBand

So, Henry said b4 he did not gain much using Icc over Gcc. I have no idea about this, anyway, I have recompiled the newest version of Gcc for now.

2. About Scotch, sorry for the misleading, I just read the "New Features" on the official website of OpenCFD, and it claims metis will be replaced with Scotch in the next release.

3. A question,
If mpi implementation is compiled with compilers other than Gnu-Gcc, but OpenFOAM is compiled with Gcc,
and,
If they are both compiled with Gnu-Gcc,
Which combination is faster?
I asked bcoz, the system mpi implementation (openmpi) is not compiled by Gnu-Gcc, but PGI's compiler.

4. i have problem installing mvapich2 which is designed for infiniband. Bcoz, several days before, the simulation of combination openmpi+OpenFOAM (both are compiled with Gcc) was so slow that I want to shoot somebody. So now I want to have a try of mpich2+openfoam (both are compiled with Gcc).

My team does not have a CFD background, I hope I could stop messing about and graduate as early as possible.

Some Updates.

1. mvapich and mvapich2 implementation failed.
   
2. starting mpich2 failed, {mpd &, mpirun -machinefile $PWD/nodelist -np ...} didn't work.
   
3. I compiled new version mpich-1.2.7p1, and it works now.
   
4. I compared the speedup with different number of processors, but the speedup is far from linear, using 8 processors, the speedup is only about 3.5, and using 16 processors, the speedup is only 9.1, ie. a sublinear speedup of 43.1%. That's very bad, isn't it?
   If you want to know further results (some figures and tables), please email me and let me know. lakeat@gmail.com
   
5. Simon's comment is of great value, much appreciated.
   Quote:

   In my case, it made a huge difference (note I'm using 1.5-dev). With the pre-compiled version, there was a small speedup at 4 or 8 processors, but far from a linear speed. When using more than 8 processors, it was slower and the clocktime was much larger than the cpu time. After being compiled on the supercomputer with the Intel compiler, the speedup is much better.

6. Next step, I am going to compare my compiled mpich and openmpi, and then using the same number of processors but on different nodes.

And thanks to everyone. :)