CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Problem with openmpi

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   October 31, 2007, 13:46
Default Hello to everybody! I'm tes
  #1
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hello to everybody!

I'm testing the new release of openFoam (1.4.1) with the openmpi libraries.

Starting a job with mpirun, I've obtain this error:

michele@enercluster:~$ mpirun -hostfile machines -np 3 a.out
bash: orted: command not found
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1164
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
[enercluster:06382] ERROR: A daemon on node 192.168.0.2 failed to start as expected.
[enercluster:06382] ERROR: There may be more information available from
[enercluster:06382] ERROR: the remote shell (see above).
[enercluster:06382] ERROR: The daemon exited unexpectedly with status 127.
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 188
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1196
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.

If I run the process only in the local machine it works!
I suppose is that a problem of PATH on the remote machines, infact executing this command:

michele@enercluster:~$ ssh node2 printenv |grep PATH
PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games

It means that when I log using ssh the openmpi are not correctly configured.

But if I log in the remote node using ssh:

# ssh node2
# echo $PATH
/opt/Fluent.Inc/bin:/home/michele/OpenFOAM/linux64/paraview-2.4.4/bin:/home/mich ele/OpenFOAM/OpenFOAM-1.4.1/src/openmpi-1.2.3/platforms/linux64GccDPOpt/bin:/hom e/michele/OpenFOAM/OpenFOAM-1.4.1/src/mico-2.3.12/platforms/linux64GccDPOpt/bin: /home/michele/OpenFOAM/linux64/j2sdk1.4.2_05/bin:/home/michele/OpenFOAM/linux64/ gcc-4.2.1/bin:/home/michele/OpenFOAM/michele-1.4.1/applications/bin/linux64GccDP Opt:/home/michele/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux64GccDPOpt:/home /michele/OpenFOAM/OpenFOAM-1.4.1/wmake:/home/michele/OpenFOAM/OpenFOAM-1.4.1/bin :/usr/local/bin:/usr/bin:/bin:/usr/games

Since it is a problem of the PATH environment variable with non-interactive ssh session.
The OpenFoam path is defined in .bashrc

Any ideas?

Thank you
Michele
mighelone is offline   Reply With Quote

Old   October 31, 2007, 23:48
Default Michele, check out this pos
  #2
Member
 
Alessandro Spadoni
Join Date: Mar 2009
Location: Atlanta, GA
Posts: 65
Rep Power: 8
gtg627e is on a distinguished road
Michele,

check out this post:

http://www.cfd-online.com/OpenFOAM_D...es/1/5473.html

I ran into a similar problem a while back.

Alessandro
gtg627e is offline   Reply With Quote

Old   November 1, 2007, 08:11
Default Hi Alessandro, Thank you fo
  #3
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hi Alessandro,

Thank you for your answer, but I've already read your post, but I suppose that my problem is different.

OpenFoam is correctly installed in every nodes (I try to run the case not in parallel in some nodes, without problem).

I guess the problem is related to the ssh non-interactive login, when I execute mpirun.
Infact during the non-interactive ssh login, my .bashrc settings file is not read, so my PATH doesn't include the PATH for mpi and OpenFoam.

I guess also that is a problem of openssh package distributed by debian, because working with other distro the .bashrc is correctly read ( I test using "ssh HOST printenv").


Michele
mighelone is offline   Reply With Quote

Old   November 2, 2007, 03:10
Default Hi Michele, There are a few
  #4
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Hi Michele,

There are a few problems that you might be experiencing.

Open-MPI needs to find its own binary path in order to boot the orted. OpenFOAM sets the variable 'OPENMPI_ARCH_PATH' to point to the openmpi installation. You can use that to help openmpi find itself. Calling mpirun with either of these should get the orte to boot:

a) $OPENMPI_ARCH_PATH/bin/mpirun ...
b) mpriun -prefix $OPENMPI_ARCH_PATH

If you try 'mpirun -help' and get the message the it can't find anything, try adding the following:

$ export OPAL_PREFIX=$OPENMPI_ARCH_PATH


Now that openmpi can boot, the remaining problem is getting the OpenFOAM environment set on the remote nodes. The nicest solution (thanks Henry) is to use foamExec to wrap the call to your application.

Attached is a modified version of foamExec in which the version is optional.
olesen is offline   Reply With Quote

Old   November 2, 2007, 05:06
Default Hi Mark, I'm trying to run
  #5
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hi Mark,

I'm trying to run a very simple parallel application (hello world!) with openmpi. If I run without -prefix option:

mpirun --hostfile machines -np 4 ./a.out

I obtain the previous error:
bash: orted: command not found
mpirun: killing job...

If I run with the option -prefix:

mpirun -prefix $OPENMPI_ARCH_PATH --hostfile machines -np 4 ./a.out

I don't receive any answer from my application.

furthermore I try to install the openmpi from my distro, installaed in canonical path (/usr/bin e /usr/lib), and the same program run without problem.

Now I will try the foamExec with native openmpi lib!

Thank you Michele
mighelone is offline   Reply With Quote

Old   November 2, 2007, 05:38
Default It might be that orte is worki
  #6
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
It might be that orte is working, but takes a long time to boot. Check if the orte is running on any of the remote nodes. I find the following alias quite useful:

alias psf='/bin/ps -e f -o user,pid,ppid,pgrp,command'


Increasing the mpirun verbosity and/or the debug-daemons might help figure out what is happening (see mpirun -help), but I haven't done this sort of thing for a long time.

Also try with an absolute path to your a.out, in case the working directory is somehow getting lost across the nodes.
olesen is offline   Reply With Quote

Old   November 2, 2007, 07:29
Default Nothing to due! I f a log i
  #7
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Nothing to due!

I f a log in the remote machine I don't have any process related to mpi.

Also giving the absolute path of a.out I don't receive any message!

I've found that the PATH during a non-interactive PATH, it is possible to set the environment variables in the file /etc/environment or in $HOME/.ssh/environment

The problem is that OpenFOAM has too many variable to set, and I don't found any way to execute the OpenFOAM script during the interactive login.
mighelone is offline   Reply With Quote

Old   November 2, 2007, 08:16
Default The problem is that OpenFOAM h
  #8
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Quote:
The problem is that OpenFOAM has too many variable to set, and I don't found any way to execute the OpenFOAM script during the interactive login.
The problem is setting *any* variables, not how many variables there are.

This is what foamExec helps you do. It provides a simple wrapper to source the requisite bashrc before executign whatever command.

In fact, we only set the FOAM environment as required and don't put it in my ~/.profile or ~/.bashrc.

Thus if I ssh to a remote host and check the OpenFOAM variables:

Eg,
$ ssh HOST printenv | grep WM_

shows nothing

but if I use foamExec:

$ ssh HOST /path/to/foamExec printenv | grep WM_

then I see about 18 OpenFOAM variables.
Does this not work for you?
olesen is offline   Reply With Quote

Old   November 2, 2007, 08:54
Default Hi Mark! At the moment I do
  #9
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hi Mark!

At the moment I don't try foamExec, but I solve the problem in this way:

# printenv > .ssh/environment

and I copied this file in all remote nodes.

In this way I define all my environment variables in the file .ssh/environment, that is read by ssh during non-interactive login.

Anyway the use of foamExec is very interesting in order to have a clean system without the several variables used by OpenFOAM.

I suppose that using foamExec I have anyway to set the variable related with openmpi in .ssh/environment, like:

OPENMPI_HOME=/home/michele/OpenFOAM/OpenFOAM-1.4.1/src/openmpi-1.2.3
OPENMPI_VERSION=1.2.3


since without these variables mpirun is not able to find the command orte in the remote machine.

Thank you
Michele
mighelone is offline   Reply With Quote

Old   November 2, 2007, 10:52
Default Hi Mark! I'm trying to use
  #10
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hi Mark!

I'm trying to use foamExec, after removing any references in .bashrc and .ssh/environment about OpenFOAM (if I well understood foamExec is able to set all the foam variables, before running any applications), but I obtain the following error:

Error : bashrc file could not be found for OpenFOAM-1.4.1

That I suppose is related to the fact that the script do not found the bashrc file of OpenFOAM.

How can I solve this inconvenient?

Michele
mighelone is offline   Reply With Quote

Old   November 2, 2007, 11:07
Default Error : bashrc file could not
  #11
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Quote:
Error : bashrc file could not be found for OpenFOAM-1.4.1
That I suppose is related to the fact that the script do not found the bashrc file of OpenFOAM.
I assume
1) that you are using the foamExec that I posted
2) that you copied foamExec to $WM_PROJECT_DIR/bin/
3) that $WM_PROJECT_DIR/.OpenFOAM-$WM_PROJECT_VERSION/ exists and that $WM_PROJECT_DIR/.OpenFOAM-$WM_PROJECT_VERSION/bashrc exists
4) that you are using the fully qualified path to foamExec -OR- you are using the -v option.

Check what foamExec is doing:

$ /bin/sh -x /path/to/foamExec /bin/true
OR
$ /bin/sh foamExec -v 1.4.1 /bin/true
olesen is offline   Reply With Quote

Old   November 3, 2007, 07:42
Default Hi Mark! 1) I'm executing y
  #12
Member
 
Michele Vascellari
Join Date: Mar 2009
Posts: 70
Rep Power: 8
mighelone is on a distinguished road
Hi Mark!

1) I'm executing your script
2) the script is not in the $WM_PROJECT_DIR/bin/ but in another place
3) These files and directories exist
4) I'm using the fully qualified path, but in the wrong place

Now I'm not at work, so I can not try your advices, Monday I will try again!

Thank you again for your help

Michele
mighelone is offline   Reply With Quote

Old   November 6, 2007, 06:11
Default Hi Mark and Michelle! I am fo
  #13
New Member
 
Christofer Ivarsson
Join Date: Mar 2009
Posts: 21
Rep Power: 8
cricke is on a distinguished road
Hi Mark and Michelle!
I am following your thread but get following message when trying to execute the following command

Command:

foamExec -v OpenFOAM-1.4.1 $OPENMPI_ARCH_PATH/bin/mpirun --hostfile machines -np 4 simpleFOAM $HOME VAFAB_multi -paralell

Message:

foamExec: access denied

1) I'm executing your script
2) the script is in the $WM_PROJECT_DIR/bin/
3) These files and directories exist
4) I'm using the -v option

When using the '$OPENMPI_ARCH_PATH/bin/mpirun ...' as you suggested the mpirun and orte is started on the host machine but not on the slave.

What am I doing wrong?

Regards

Christofer Ivarsson
cricke is offline   Reply With Quote

Old   November 6, 2007, 06:21
Default Message: foamExec: access de
  #14
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Quote:
Message:
foamExec: access denied
What about the next point?
5) Is foamExec readable and executable by the users (eg, chmod 0755).
olesen is offline   Reply With Quote

Old   November 6, 2007, 07:37
Default Hi and thanks for your instant
  #15
New Member
 
Christofer Ivarsson
Join Date: Mar 2009
Posts: 21
Rep Power: 8
cricke is on a distinguished road
Hi and thanks for your instant reply!


The file is perfectly readable and saved as an application-file. This is what happends when trying to execute foamExec only

a403518@ENEPCST75:~$ foamExec
bash:home/a403518/OpenFOAM/OpenFOAM-1.4.1/bin/foamExec: Access denied

Obviously its all about environment variables, right? I get the same result even with no extra environment variables added in the .bashrc

Shall I add som kind of path in the .bashrc to point out the foamExec?

/Chris
cricke is offline   Reply With Quote

Old   November 6, 2007, 07:56
Default Hi Christofer, With "Access
  #16
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Hi Christofer,

With "Access Denied", I still suspect some sort of file permissions problem. Check that the rest of your OpenFOAM installation is readable.
(eg, chmod -v -R a+rX ...). Note use large 'X' and not small 'x' in the chmod command.

Recheck what foamExec is doing:
$ /bin/sh -x /path/to/foamExec /bin/true
OR
$ /bin/sh foamExec -v 1.4.1 /bin/true


When I issue foamExec without any arguments, I get the message "no application specified" and then the usage as per the -help option.
olesen is offline   Reply With Quote

Old   November 6, 2007, 09:12
Default ok, sorry for me being a nut h
  #17
New Member
 
Christofer Ivarsson
Join Date: Mar 2009
Posts: 21
Rep Power: 8
cricke is on a distinguished road
ok, sorry for me being a nut head but do you want me to write those commands in the terminal because that is what I did and here is the results.

a403518@ENEPCST75:~$ /bin/sh -x /patch/to/foamExec /bin/true
/bin/sh: Can't open /patch/to/foamExec
a403518@ENEPCST75:~$ bin/sh foamExec -v 1.4.1 /bin/true
bash: bin/sh: File does not exist

I have no restricted access to my OpenFOAM directory and OpenFOAM runs smoothly. Do you still think its a permission problem? When I run the chmod command it suddenly changed the permission to my OpenFOAM-directory to locked so I had to change that back again.

/Chris
cricke is offline   Reply With Quote

Old   November 6, 2007, 10:04
Default There was a typo there: $ /
  #18
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
There was a typo there:

$ /bin/sh -x /the/path/to/foam/bin/on/your/machine/foamExec

I don't know the path to foam on your machine.
olesen is offline   Reply With Quote

Old   November 6, 2007, 10:23
Default Thanks for you being patient w
  #19
New Member
 
Christofer Ivarsson
Join Date: Mar 2009
Posts: 21
Rep Power: 8
cricke is on a distinguished road
Thanks for you being patient with me. Following happends

Recheking what foamExec is doing when:

a403518@ENEPCST75:~$ /bin/sh foamExec -v 1.4.1 $HOME/OpenFOAM/OpenFOAM-1.4.1/bin/true
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.bashrc
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.OpenFOAM-1.4.1/apps/ensightFoam/bashrc
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.OpenFOAM-1.4.1/apps/paraview/bashrc
exec: 141: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/bin/true: not found

OR

403518@ENEPCST75:~$ /bin/sh -x $HOME/OpenFOAM/OpenFOAM-1.4.1/ bin/true
a403518@ENEPCST75:~$

The latter seem to work since it doesnt protest in the command window. However the first command seem to find all env-variables except 'true'. I must tell that I have already added the first three:
.../.bashrc
.../paraview/basrc
.../ensightFOAM/bashrc

in my .bashrc so that they set automatically when opening a terminal window.

So whats up next?
I get contact with my slave node and I enter the password. Then nothing else happends in the command window. The mpirun and orted start but are listed as sleeping processes. The solver never starts.

/Chris
cricke is offline   Reply With Quote

Old   November 7, 2007, 03:28
Default Hi Christofer, I really tho
  #20
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 777
Rep Power: 18
olesen will become famous soon enough
Hi Christofer,

I really thought that absolutely every Unix machine had '/bin/true', but if not you can always test with a command such as 'hostname' which normally lives under '/bin' but if your system is really strange it might be under '/usr/bin' or I don't know where else.

You *do* however need to setup ssh to connect without passwords, otherwise you can forget about the rest. There must be something about this in the open-mpi FAQ. If not, check the mpich FAQ.

/mark
olesen is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problems with Fedora 9 and OpenMPI harly OpenFOAM Running, Solving & CFD 11 May 3, 2009 04:18
SnappyHexMesh in parallel openmpi wikstrom OpenFOAM Mesh Utilities 7 November 24, 2008 10:52
Problems using local openmpi stephan OpenFOAM Installation 1 December 5, 2007 19:01
OpenMPI performance vega OpenFOAM Running, Solving & CFD 13 November 27, 2007 02:28
OpenFOAM 14 with OpenMPI 12 fhy OpenFOAM Installation 0 July 12, 2007 18:12


All times are GMT -4. The time now is 03:23.