[lakeat]

Dear All,

I am struggling with porting OpenFoam to my IRIX with "-parallel" option on. That means I can run the solvers without "-parallel" very well, but as I mpirun my solver, it failed with annoying coredump. The output is at the bottom of my post.

what I don't understand is these lines in argList.C:
-------------------------------------------------

wordList machinesInfo(1);
machinesInfo[0] = hostName();

labelList pids(1);
pids[0] = pid();

if (parRunControl_.parRun())
{
if (Pstream::master())
{
machinesInfo.setSize(Pstream::nProcs());
pids.setSize(Pstream::nProcs());

// From all slaves
for
(
int slave=Pstream::firstSlave();
slave<=Pstream::lastSlave();
slave++
)
{
label slavei = slave - Pstream::masterNo();
IPstream fromSlave(slave);
fromSlave >> machinesInfo[slavei] >> pids[slavei];

}
else
{
OPstream toMaster(Pstream::masterNo(), 0, false);
toMaster << machinesInfo[0] << pids[0];
}
}
}
-------------------------------------------------

1. How can pids[slavei] get 4 (in my case) different pid()? The reason I raise the question is because I found my pids[1] is zero, and my machinesInfo[slavei] is empty! that's strange.
2. Is there a way to store the 3 slave (in my case) pids? (they are : 513638,513610,513608.)

Please help! Thanks in advance!

Daniel

--------------------------------------------------
bash-2.05$ mpirun -np 4 `which icoFoam` $FOAM_RUN/tutorials/icoFoam cavity -parallel
MPI Pstream initialized with:
floatTransfer : 1
nProcsSimpleSum : 0
scheduledTransfer : 0

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.4.1-dev |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam /disk4/usr/people/student3/OpenFOAM/student3-1.4.1-dev/run/tutorials/icoFoam cavity -parallel
[0] Date : Apr 28 2008
[0] Time : 16:44:15
[0] Host : onyx-bridge
[0] PID : 513629
[3] Date : Apr 28 2008
[3] Time : 16:44:15
[3] Host : onyx-bridge
[3] PID : 513638
[1] Date : Apr 28 2008
[1] Time : 16:44:15
[1] Host : onyx-bridge
[1] PID : 513610
[2] Date : Apr 28 2008
[2] Time : 16:44:15
[2] Host : onyx-bridge
[2] PID : 513608
MPI: Program /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam, Rank 3, Process 513638 received signal SIGBUS(10)


MPI: --------stack traceback-------
PC: 0x5ddb100 MPI_SGI_stacktraceback in /usr/lib32/libmpi.so
PC: 0x5ddb544 first_arriver_handler in /usr/lib32/libmpi.so
PC: 0x5ddb7d8 slave_sig_handler in /usr/lib32/libmpi.so
PC: 0xfaee79c _sigtramp in /usr/lib32/libc.so.1
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x60204334 in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x42d432c (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
6 0x60197ac8 abs
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x6020374c in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
7 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x42d3744 (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x602064bc in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x42d64b4 (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x601ee308 in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x42be300 (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x601c262c in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x4292624 (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
libexc(513638): ERROR __exc_get_fde_name: Couldn't find any name at 0x601b0904 in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
0 0x60197ac8 abs
(a lot of messages the same as above)
(......)
115 0x60197ac8 abs
PC: 0x42808fc (null) in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/lib/sgiN32GccDPDebug/libO penFOAM.so
PC: 0x1002aeec main in /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam


MPI: dbx version 7.3.3 (78517_Dec16 MR) Dec 16 2001 07:45:22
MPI: Process 513638 (icoFoam) stopped at [__waitsys:24 +0x8,0xfa53338]
MPI: Source (of /xlv42/6.5.23m/work/irix/lib/libc/libc_n32_M4/proc/waitsys.s) not available for Process 513638
MPI: > 0 __waitsys(0x0, 0x7d662, 0x7ffb5080, 0x3, 0x0, 0x7ffb4ec8, 0x0, 0xf8) ["/xlv42/6.5.23m/work/irix/lib/libc/libc_n32_M4/proc/waitsys.s":24, 0xfa53338]
MPI: 1 _system(0x7ffb5150, 0x7d662, 0x7ffb5080, 0x3, 0x0, 0x7ffb4ec8, 0x0, 0xf8) ["/xlv42/6.5.23m/work/irix/lib/libc/libc_n32_M4/stdio/system.c":116, 0xfa5f868]
MPI: 2 MPI_SGI_stacktraceback(0x0, 0x7d662, 0x7ffb5080, 0x3, 0x0, 0x7ffb4ec8, 0x0, 0xf8) ["/xlv4/mpt/1.8/mpi/work/4.3/lib/libmpi/libmpi_n32_M4/adi/sig.c":242, 0x5ddb268]
MPI: 3 first_arriver_handler(0xa, 0x71756974, 0x7ffb5080, 0x3, 0x0, 0x7ffb4ec8, 0x0, 0xf8) ["/xlv4/mpt/1.8/mpi/work/4.3/lib/libmpi/libmpi_n32_M4/adi/sig.c":445, 0x5ddb544]
MPI: 4 slave_sig_handler(0xa, 0x7d662, 0x7ffb5080, 0x3, 0x0, 0x7ffb4ec8, 0x0, 0xf8) ["/xlv4/mpt/1.8/mpi/work/4.3/lib/libmpi/libmpi_n32_M4/adi/sig.c":542, 0x5ddb7e0]
MPI: 5 _sigtramp(0x0, 0x7d662, 0x0, 0x3, 0x0, 0x0, 0x0, 0xf8) ["/xlv42/6.5.23m/work/irix/lib/libc/libc_n32_M4/signal/sigtramp.s":71, 0xfaee79c]
MPI: 6 Foam::IPstream::readFromBuffer(this = <illegal>, t = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/db/IOstrea ms/Pstreams/IPread.C":2, 0x42d4334]
MPI: 7 Foam::IPstream::read(this = <illegal>, l = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/db/IOstrea ms/Pstreams/IPread.C":27, 0x42d3744]
MPI: 8 Foam::IPstream::read(this = <illegal>, t = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/db/IOstrea ms/Pstreams/IPreadToken.C":4
MPI: More (n if no)?, 0x42d64b4]
MPI: 9 Foam::token::token(this = <illegal>, is = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/db/IOstrea ms/token/tokenIO.C":4, 0x42be300]
MPI: 10 <unknown>(is = <illegal>, L = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/global/arg List/argList.C":40, 0x4292624]
MPI: 11 <unknown>(this = <illegal>, argc = <illegal>, argv = <illegal>, checkArgs = <illegal>, checkOpts = <illegal>) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/src/OpenFOAM/global/arg List/argList.C":66, 0x42808fc]
MPI: 12 ::main(The debugger has detected a dbxpcs server error (Internal Consistency Error).
MPI: dbxpcs server attempting to continue
MPI: ) ["/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/solvers/in compressible/icoFoam/icoFoam.C":0, 0x1002aeec]
MPI: 13 __start() ["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177, 0x1002a668]

MPI: -----stack traceback ends-----
MPI: Program /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam, Rank 3, Process 513638: Dumping core on signal SIGBUS(10) into directory /disk4/usr/people/student3/OpenFOAM/student3-1.4.1-dev/run/tutorials/icoFoam
MPI: Program /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam, Rank 1, Process 513610: Core dump on signal SIGBUS(10) suppressed.
MPI: Program /disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32Gc cDPDebug/icoFoam, Rank 2, Process 513608: Core dump on signal SIGBUS(10) suppressed.
MPI: MPI_COMM_WORLD rank 1 has terminated without calling MPI_Finalize()
MPI: aborting job
MPI: Received signal 9

[lakeat]

Now, I found the problem is around these line in argList.C
from line 339
--------------------------
else
{
// Collect the master's argument list
IPstream fromMaster(Pstream::masterNo());
fromMaster >> args_ >> options_;
}
--------------------------
obviously, the program fail at line
fromMaster >> args_ >> options_;

Could anybody give me some help on what does these lines mean, in Istream.H.
--------------------------
typedef Istream& (*IstreamManip)(Istream&);

//- operator>> handling for manipulators without arguments
inline Istream& operator>>(Istream& is, IstreamManip f)
{
return f(is);
}

//- operator>> handling for manipulators without arguments
inline Istream& operator>>(Istream& is, IOstreamManip f)
{
f(is);
return is;
}
-------------------------------

I doubt whether there is restriction on the length of filename on my IRIX, for when i insert
---------------
Sout << args() << endl;
---------------
like this
---------------
else
{
// Collect the master's argument list
Sout << args() << endl;
IPstream fromMaster(Pstream::masterNo());
fromMaster >> args_ >> options_;
}
----------------
output is
/disk4/usr/people/student3/OpenFOAM/OpenFOAM-1.4.1-dev/applications/bin/sgiN32SP Debug/icoFoam

is this too long?

Any hints? Thanks

Daniel

[mattijs]

(following up from http://www.cfd-online.com/cgi-bin/OpenFOAM_Discus/show.cgi?tpc=126&post=23069#POST230 69)

- write a non-foam c++ mpi test program which does mpi_init and a few send/receive.
- put a few print stmts into Pstream/mpi/*.C to show you what is being received/sent or link OF with a tracing library (mpe?)
- begin() returns a pointer to the underlying storage.

[lakeat]

> write a non-foam c++ mpi test program which does mpi_init and a few send/receive.
I did it before and everying works perfect.

> put a few print stmts into Pstream/mpi/*.C to show you what is being received/sent or link OF with a tracing library (mpe?)
I am doing that!

Thank you, Mattijs!
Daniel

[lakeat]

Hi Mattijs!

I found something new. Look at the error messages I got!
......
MPI: dbx version 7.3.3 (78517_Dec16 MR) Dec 16 2001 07:45:22
MPI: Process 513638 (icoFoam) stopped at [__waitsys:24 +0x8,0xfa53338]
MPI: Source (of /xlv42.../waitsys.s) not available for Process 513638
MPI: > 0 __waitsys(0x0...........................sys.s":24, 0xfa53338]
MPI: 1 _system(0x7ffb5...................dio/system.c":116, 0xfa5f868]
MPI: 2 MPI_SGI_stacktr......................adi/sig.c":242, 0x5ddb268]
MPI: 3 first_arriver_h..............................c":44 5, 0x5ddb544]
MPI: 4 slave_sig_handl..........bmpi_n32_M4/adi/sig.c":542, 0x5ddb7e0]
MPI: 5 _sigtramp(0x0, ...............signal/sigtramp.s":71, 0xfaee79c]
MPI: 6 Foam::IPstream::readFromBuffer(this = <illegal>, t = <illegal>) [".../src/OpenFOAM/db/IOstrea ms/Pstreams/IPread.C":2, 0x42d4334]
MPI: 7 Foam::IPstream::read(this = <illegal>, l = <illegal>) [".../src/OpenFOAM/db/IOstreams/Pstreams/IPread.C":27, 0x42d3744]
MPI: 8 Foam::IPstream::read(this = <illegal>, t = <illegal>) [".../src/OpenFOAM/db/IOstreams/Pstreams/IPreadToken.C":4
MPI: 9 Foam::token::token(this = <illegal>, is = <illegal>) [".../src/OpenFOAM/db/IOstreams/token/tokenIO.C":4, 0x42be300]
MPI: 10 <unknown>(is = <illegal>, L = <illegal>) [".../src/OpenFOAM/global/arg List/argList.C":40, 0x4292624]
MPI: 11 <unknown>(this = <illegal>, argc = <illegal>, argv = <illegal>, checkArgs = <illegal>, checkOpts = <illegal>) [".../src/OpenFOAM/global/arg List/argList.C":66, 0x42808fc]
MPI: 12 ::main(The debugger has detected a dbxpcs server error (Internal Consistency Error).
MPI: dbxpcs server attempting to continue
MPI: ) [".../applications/solvers/in compressible/icoFoam/icoFoam.C":0, 0x1002aeec]
MPI: 13 __start() ["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177, 0x1002a668]
......

1st,
from "MPI: 12" to "MPI: 6", it seems that the "Foam::IPstream::readFromBuffer" (in ./src/OpenFOAM/IOstreams/Pstreams/IPread.C) is the arch-criminal, I GUESS!
So, I tried to print out the "bufPosition_" & "buf_[bufPosition_]" as the following.
Note, I used "Sout" instead of "Info", for the latter didn't work in slave process, that's strange.
|......
|template<class>
|inline void IPstream::readFromBuffer(T& t)
|{
|````Sout<< "bufPosition_````````:" << bufPosition_ << nl````````````(Output is 1)
|````````<< "buf_[bufPosition_]``:" << buf_[bufPosition_] << endl;```(Output is nothing! This is the same as buf_.begin() and ||||buf_.end())
|````t = reinterpret_cast<t&>(buf_[bufPosition_]);```````````````````(ERROR!!!!!!!!!!!)
|````Sout<< "blahblahblahblahblahblahblahblahblah " << endl;```````` (Die with NO output)
|````bufPosition_ += sizeof(T);
|````checkEof();
|
|````//readFromBuffer(&t, sizeof(T));
|}
|......

2nd,
In error messages, there are so many "illegal"s, though everything works fine with single CPU of IRIX.
So, I have 2 questions concerning line 87-106 in ./src/Pstream/mpi/Pstream.C

|......
|#```ifndef SGIMPI
|````string bufferSizeName = getEnv("MPI_BUFFER_SIZE");
|
|````if (bufferSizeName.size())
|````{
|````````int bufferSize = atoi(bufferSizeName.c_str());
|
|````````if (bufferSize)
|````````{
|````````````MPI_Buffer_attach(new char[bufferSize], bufferSize);
|````````}
|````}
|````else
|````{
|````````FatalErrorIn("Pstream::init(int& argc, char**& argv)")
|````````````<< "Pstream::init(int& argc, char**& argv) : "
|````````````<< "environment variable MPI_BUFFER_SIZE not defined"
|````````````<< Foam::abort(FatalError);
|````}
|#```endif
|......

a) what if I use SGIMPI now, is that SGIMPI don't need MPI_Buffer_attach() anymore?
b) in ./src/malloc, which one should I use, bsdmalloc or fbsdmalloc?


Thanks and Best Regards, Daniel

[mattijs]

- you can't use Info/Pout yet. Isn't initialised yet.

- check whether SGIMPI still does not like MPI_Buffer_attach. That exclusion is from long long time ago.

[lakeat]

Hello Majjijs,

First, to OpenFOAM-1.5, congratulations!

I am curious!
Is the new version being tested on SGI?
Or the endeavor has been given up? For It seems the wmake/rules/sgi* have been removed!!

Another question where can I get OpenFOAM-1.5-dev?

Regards,
Daniel

[mattijs]

Hi Daniel,

no we've not tested on SGI. Where possible we've split the OS specifics off to a separate library (OSspecific) to make porting easier.