CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Message truncated, error stack: MPIDI_CH3U_Receive_data_found

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   October 27, 2013, 17:05
Default Message truncated, error stack: MPIDI_CH3U_Receive_data_found
  #1
Senior Member
 
Vishal Nandigana
Join Date: Mar 2009
Location: Champaign, Illinois, U.S.A
Posts: 206
Rep Power: 9
nandiganavishal is on a distinguished road
Hi Brunos,

I was trying to install OF 2.0.0 in CentOS version 4.x. It did not work. Anyway, I think I would update the OS and install it in the future.

However, I have another question regarding running codes in parallel in OF. It appears that my solver runs fine in serial mode, however, when I run it in solver, I received the following error message.

Fatal error in MPI_Recv:
Message truncated, error stack:
MPIDI_CH3U_Receive_data_found(257): Message from rank 2 and tag 1 truncated; 8 bytes received but buffer size is 4


I tried to search for any similar errors encountered by the OF users before, but could not find any good suggestion. Please let me know what is the source for this error and how to fix this issue.

Thanks

Regards,
Vishal
nandiganavishal is offline   Reply With Quote

Old   October 28, 2013, 03:47
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Vishal,

I suspect that you need to find the correct environment variable for setting the buffer size for the MPI toolbox to use. For example, OpenFOAM sets for Open-MPI the variable "MPI_BUFFER_SIZE": https://github.com/OpenFOAM/OpenFOAM...ttings.sh#L580

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 28, 2013, 11:28
Default
  #3
Senior Member
 
Vishal Nandigana
Join Date: Mar 2009
Location: Champaign, Illinois, U.S.A
Posts: 206
Rep Power: 9
nandiganavishal is on a distinguished road
Hi Brunos,

Could you provide more details on this one. I am not that familiar on how to identify the environment variables to set the buffer size. I did look at the settings.sh file and the MPI_BUFFER_SIZE is set similar to the link you had mentioned. It would be great if you could help me out in this one.

Thanks

Regards,
Vishal
nandiganavishal is offline   Reply With Quote

Old   October 28, 2013, 17:12
Default
  #4
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Vishal,

I've moved this line of conversation from OpenFOAM-1.7.0 for CentOS/RHEL/SL 5.x 64bit released to this new thread, because the other one refers to installing OpenFOAM, not regarding running it in parallel

As for your problem, I need to know a few things:
  1. On which Linux Distribution are you trying to run OpenFOAM in parallel?
  2. Which OpenFOAM version are you using for this?
  3. Which MPI toolbox is being used with OpenFOAM? To ascertain this:
    1. Check which MPI is chosen on OpenFOAM's environment:
      Code:
      echo $FOAM_MPI
    2. Check which mpirun is being found:
      Code:
      which mpirun
      ls -l $(which mpirun)
    3. Check which version of MPI it's being used:
      Code:
      mpirun --version
  4. What exact command are you using for launching the application in parallel?
Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 29, 2013, 12:38
Default
  #5
Senior Member
 
Vishal Nandigana
Join Date: Mar 2009
Location: Champaign, Illinois, U.S.A
Posts: 206
Rep Power: 9
nandiganavishal is on a distinguished road
Hi Brunos,

Below is the reply for all your questions.

1. Linux distribution:

Linux taubh1 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 11:13:47 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux
Scientific Linux release 6.1 (Carbon)

2. OpenFoam version: OpenFoam-1.7.1

3. Which MPI toolbox is being used with OpenFOAM?

Command: echo $FOAM_MPI (did not work)

Command used: echo $FOAM_MPI_LIBBIN
message: /home/nandiga1/OpenFOAM/OpenFOAM-1.7.1/lib/linux64GccDPOpt/mvapich2-1.6-gcc+ifort

3.2. Check which mpirun is being found:

which mpirun
/usr/local/mvapich2-1.6-gcc+ifort/bin/mpirun

ls -l $(which mpirun)
lrwxrwxrwx 1 394298 394298 13 2011-11-18 16:53 /usr/local/mvapich2-1.6-gcc+ifort/bin/mpirun -> mpiexec.hydra

3.3. Check which version of MPI it's being used:

mpirun --version

HYDRA build details:
Version: 1.6rc3
Release Date: unreleased development copy
CC: gcc -fpic
CXX: g++ -fpic
F77: ifort -fpic
F90: ifort -fpic
Configure options: '--prefix=/usr/local/mvapich2-1.6-gcc+ifort' 'CC=gcc -fpic' 'CXX=g++ -fpic' 'F77=ifort -fpic' 'F90=ifort -fpic' 'FC=ifort -fpic' '--with-mpe' '--enable-sharedlibs=gcc' '--disable-checkerrors' '--with-atomic-primitives=auto_allow_emulation' 'CFLAGS= -DNDEBUG -O2' 'LDFLAGS= ' 'LIBS= -lpthread -libverbs -libumad -ldl -lrt ' 'CPPFLAGS= -I/usr/local/src/mvapich/mvapich2-1.6/src/openpa/src -I/usr/local/src/mvapich/mvapich2-1.6/src/openpa/src'
Process Manager: pmi
Launchers available: ssh rsh fork slurm ll lsf sge none persist
Binding libraries available: hwloc plpa
Resource management kernels available: none slurm ll lsf sge pbs
Checkpointing libraries available:
Demux engines available: poll select

3.4 What exact command are you using for launching the application in parallel?

#PBS -q cse
#PBS -l nodes=1pn=12
#PBS -l walltime=60:30:00
#PBS -j oe
#PBS -o simout
#PBS -N 2D_circular
cd ${PBS_O_WORKDIR}
module load mvapich2/1.6-gcc+ifort
mpiexec -np 12 circularFoam_full -parallel

Hope this would provide you some idea regarding the problem.

Thanks

Regards,
Vishal
nandiganavishal is offline   Reply With Quote

Old   November 2, 2013, 17:24
Default
  #6
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Vishal,

OK, I got a better idea of the system you've got, but no clear notion as to why this error occurs.

Some searching online gave me the indication that it could be a memory limitation problem on the machines themselves. In other words, perhaps the mesh is too big for the machines you want to use.

Another indication was that there is no way to control the buffer size on mvapich2.

I suggest that you do a basic communication test on the cluster, following the instructions given here on how to test if MPI is working: post #4 of "openfoam 1.6 on debian etch", and/or post #19 of "OpenFOAM updates"

Then try to run one of OpenFOAM's tutorials in parallel, such as the tutorial "multiphase/interFoam/laminar/damBreak".

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Guide: Getting Started with the CFD Online Discussion Forums pete Site Help, Feedback & Discussions 4 September 27, 2013 20:49
single directional message transmit in parallel model su_junwei OpenFOAM Programming & Development 1 December 17, 2009 08:00
Error message in CFX-10 CFDworker CFX 2 October 12, 2007 07:23
error message susan CD-adapco 0 August 17, 2007 00:27
MPI Message truncated josef OpenFOAM Running, Solving & CFD 1 January 9, 2006 07:29


All times are GMT -4. The time now is 13:24.