CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

PyFoam killing my SSH sessions

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By gschaider

Reply
 
LinkBack Thread Tools Display Modes
Old   April 25, 2014, 11:23
Default PyFoam killing my SSH sessions
  #1
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Dear Foamers,

I recently decided to switch to PyFoam to run and monitor my cases. However, I encountered an annoying issue: some PyFoam functions seem to kill my SSH sessions. Here's the kind of output I get:
Code:
<some output>
patch 0 (start: 19800 size: 50) name: inlet
patch 1 (start: 19850 size: 50) name: outlet
patch 2 (start: 19900 size: 50) name: lowerWall
patch 3 (start: 19950 size: 50) name: upperWall
patch 4 (start: 20000 size: 200) name: cylinderWalls
patch 5 (start: 20200 size: 10000) name: back
patch 6 (start: 30200 size: 10000) name: front

End

Killing PID 29668
Connection to hpc4 closed by remote host.
Connection to hpc4 closed.
Sometime it happens, sometimes it doesn't. Any idea of what could be the cause of this behaviour?
leroyv is offline   Reply With Quote

Old   April 25, 2014, 11:51
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,523
Blog Entries: 36
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Vincent,

That's a curious occurrence. A few questions:
  1. Which Python version are you using?
  2. Can you give specific examples of the commands you're using?
  3. Are you're using any scripts for launching the PyFoam scripts?
  4. Are the simulations being launched by a job scheduler?
Best regards,
Bruno
__________________
___
I'll be at OFW11 in Portugal
wyldckat is offline   Reply With Quote

Old   April 25, 2014, 15:27
Default
  #3
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,920
Rep Power: 41
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by leroyv View Post
Dear Foamers,

I recently decided to switch to PyFoam to run and monitor my cases. However, I encountered an annoying issue: some PyFoam functions seem to kill my SSH sessions. Here's the kind of output I get:
Code:
<some output>
patch 0 (start: 19800 size: 50) name: inlet
patch 1 (start: 19850 size: 50) name: outlet
patch 2 (start: 19900 size: 50) name: lowerWall
patch 3 (start: 19950 size: 50) name: upperWall
patch 4 (start: 20000 size: 200) name: cylinderWalls
patch 5 (start: 20200 size: 10000) name: back
patch 6 (start: 30200 size: 10000) name: front

End

Killing PID 29668
Connection to hpc4 closed by remote host.
Connection to hpc4 closed.
Sometime it happens, sometimes it doesn't. Any idea of what could be the cause of this behaviour?
That is not a lot you're giving us to work with. What SHOULD happen? (in addition to the stuff that Bruno asked)

I've never seen that kind of behaviour and can only guess (although most guesses say "it's not my (PyFoam) fault but a weird thing where the termination signal of the thread that runs the OF-command". But that is a REALLY wild guess)
Bernhard likes this.
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   April 27, 2014, 06:07
Default
  #4
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Greetings Vincent,

That's a curious occurrence. A few questions:
  1. Which Python version are you using?
  2. Can you give specific examples of the commands you're using?
  3. Are you're using any scripts for launching the PyFoam scripts?
  4. Are the simulations being launched by a job scheduler?
Best regards,
Bruno
Dear Bruno,

Thank you for answering so quickly. Here are the answers to your questions:
  1. I'm using Python 2.7.3
  2. An example of the kind of script I use is included below
  3. I am directly launching my Allrun.py script from an interactive ZSH session (over SSH)
  4. I don't use a job scheduler
Vincent

Allrun.py
Code:
#!/usr/bin/python

# Run script
# Requires pyFoam

import os, subprocess, configobj
from PyFoam.Applications.ClearCase import ClearCase
from PyFoam.Applications.Runner import Runner
from PyFoam.Applications.FromTemplate import FromTemplate

def touch(path):
    with open(path, 'a'):
        os.utime(path, None)

def main():
    """
    Steps for the run:
    - Cleanup the case (remove time directories)
    - Build the mesh
    - Run the createPatch tool
    - Run the mapFields utility (import average fields)
    - Run the decomposePar tool
    - Run the application
    - Run the postprocessing programs (reconstructPar, sample, foamCalc and foamCalcEx)
    - Generate the .foam files for visualization with ParaView
    """

    nProc = 2
    rootPath = os.getcwd()

    # Load case parameters
    CaseParameters = configobj.ConfigObj("CaseParameters.ini")

    # Cleanup
    ClearCase()

    # Build dictionaries
    os.chdir(os.path.join(rootPath, "constant/polyMesh/"))
    subprocess.call(["python", "blockMeshDict.py"])
    os.chdir(rootPath)

    os.chdir(os.path.join(rootPath, "system/"))
    subprocess.call(["python", "createPatchDict.py"])
    os.chdir(rootPath)

    for dictName in CaseParameters['Dictionaries']:
        FromTemplate(args=[ dictName, CaseParameters['Dictionaries'][dictName] ])

    # Move resulting files in 0/templates to their runtime location
    os.rename("0/templates/TTilde", "0/TTilde")

    # Build mesh
    Runner(args=["blockMesh"])

    # Run the createPatch utility
    Runner(args=["createPatch", "-overwrite"])

    # Map input average fields
    Runner(args=[
                "mapFields",
                "input/average"
                ])

    # Map input velocity deviations
    Runner(args=[
                "mapFields", 
                "-consistent",
                "input/cell"
                ])
    

    # Decompose
    Runner(args=[
                "decomposePar", 
                "-force"
                ])

    # Run application
    Runner(args=[
                "--proc=%d"%nProc,
                "thermoDownscalingFoam",
                ])

    # Reconstruct if necessary
    Runner(args=["reconstructPar"])

    # Create visualization files if necessary
    touch("case.foam")


if __name__ == "__main__":
    main()
Attached Files
File Type: txt Allrun.py.txt (2.2 KB, 3 views)
leroyv is offline   Reply With Quote

Old   April 27, 2014, 06:13
Default
  #5
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by gschaider View Post
That is not a lot you're giving us to work with. What SHOULD happen? (in addition to the stuff that Bruno asked)

I've never seen that kind of behaviour and can only guess (although most guesses say "it's not my (PyFoam) fault but a weird thing where the termination signal of the thread that runs the OF-command". But that is a REALLY wild guess)
Dear Bernhard,

Thank you for your answer. In the example output I posted, the killing happens right at the end of the execution of the blockMesh utility using a Runner object (see the Allrun.py script posted above). The script should then proceed with the execution of various utilities and a solver.

Vincent
leroyv is offline   Reply With Quote

Old   April 27, 2014, 19:11
Default
  #6
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,920
Rep Power: 41
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by leroyv View Post
Dear Bernhard,

Thank you for your answer. In the example output I posted, the killing happens right at the end of the execution of the blockMesh utility using a Runner object (see the Allrun.py script posted above). The script should then proceed with the execution of various utilities and a solver.

Vincent
I had a look: PyFoam only kills a run if it receives a keyboard-interrupt (=user pressed Ctrl-C or similar). So it seems that something happened that "looks like a keyboard interrupt". But I have no idea what that could be in your case
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   April 28, 2014, 14:38
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,523
Blog Entries: 36
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

Quote:
Originally Posted by gschaider View Post
I had a look: PyFoam only kills a run if it receives a keyboard-interrupt (=user pressed Ctrl-C or similar). So it seems that something happened that "looks like a keyboard interrupt". But I have no idea what that could be in your case
There is a feature in SSH connections for keeping a connection alive, but I'm not certain how it works. One of the possible strategies might be similar to a keyboard character being sent periodically.

I'm not familiar enough with Python to know this, but when using bash scripts, it's possible to use the "-e" set option:
Code:
set -e
that defines that the script should terminate when an error occurs. It can be turned off later in the script with:
Code:
set +e
I can only guess that Python might have a similar feature, i.e. don't quit until there are any major errors.

Best regards,
Bruno
__________________
___
I'll be at OFW11 in Portugal
wyldckat is offline   Reply With Quote

Old   April 29, 2014, 05:35
Default
  #8
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,920
Rep Power: 41
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by wyldckat View Post
Greetings to all!


There is a feature in SSH connections for keeping a connection alive, but I'm not certain how it works. One of the possible strategies might be similar to a keyboard character being sent periodically.

I'm not familiar enough with Python to know this, but when using bash scripts, it's possible to use the "-e" set option:
Code:
set -e
that defines that the script should terminate when an error occurs. It can be turned off later in the script with:
Code:
set +e
I can only guess that Python might have a similar feature, i.e. don't quit until there are any major errors.

Best regards,
Bruno
The problem is not the SSH-connection closing (otherwise we wouldn't see the "Killing"-message. Which comes from PyFoam). The problem is that PyFoam thinks that somebody pressed Ctrl-C and decides to kill the process. That information comes from the Thread-subsystem so there is not much I can do about. Probably the same one who passes out this wrong information also passes it to the running shell and that closes the connection.

One thing: it seems that the output from PyFoam messes up some terminals. What you could try is to add the "--silent"-option to the Runner calls. This will print nothing to the terminal but you should still have the log-files (don't use "--progress" because it tends to make this worse: some terminals have problem with '\r')

Other option is to go to PyFoam.Execution.BasicRunner.py and add print-statemtents before self.run.interrupt() to find out how the interrupt got there (there should also be traces in ~/pyFoam/log/general and the PyFoamState.TheState in the case should read "Interrupted" if PyFoam thinks that someone Ctrl-C'ed it (but that is only diagnostic and I wouldn't know how to fix this)
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   April 29, 2014, 05:52
Default
  #9
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Thank you for your advice, gentlemen. I will try to do what Bernhard suggests ASAP and I'll come back if I make any progress.

Regards,

Vincent
leroyv is offline   Reply With Quote

Old   May 5, 2014, 07:55
Default Update
  #10
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by gschaider View Post
One thing: it seems that the output from PyFoam messes up some terminals. What you could try is to add the "--silent"-option to the Runner calls. This will print nothing to the terminal but you should still have the log-files (don't use "--progress" because it tends to make this worse: some terminals have problem with '\r')
Dear Bernhard,

I suppressed the terminal output using the --silent option. Killings still occurred. I noticed that they seem to happend preferably at the end of the execution of utilities (all of them) and quite randomly. I tried to run the utilities using the UtilityRunner class and killings stopped occurring.

However, I still get errors like:
Code:
Getting LinuxMem: [Errno 2] No such file or directory: '/proc/27467/status'
I browsed the forum and found this post: PyFoam . Might this be related to my problem?
leroyv is offline   Reply With Quote

Old   May 5, 2014, 12:54
Default
  #11
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,920
Rep Power: 41
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by leroyv View Post
Dear Bernhard,

I suppressed the terminal output using the --silent option. Killings still occurred. I noticed that they seem to happend preferably at the end of the execution of utilities (all of them) and quite randomly. I tried to run the utilities using the UtilityRunner class and killings stopped occurring.

However, I still get errors like:
Code:
Getting LinuxMem: [Errno 2] No such file or directory: '/proc/27467/status'
I browsed the forum and found this post: PyFoam . Might this be related to my problem?
Hm. UtilityRunner from PyFoam.Applications or PyFoam.Execution? In the later case one big difference is that NO server process is constructed automatically. What you could try is whether the --no-server-process-option makes a difference for the PyFoam.Applications.Runner-class

Anyway: If you're doing scripts PyFoam.Execution.UtilitiyRunner might be the better choice anyway (the PyFoam.Applications-classes are mainly for quickly "translating" what you did on the shell to a script)
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   May 9, 2014, 10:44
Smile Solution found
  #12
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by gschaider View Post
Hm. UtilityRunner from PyFoam.Applications or PyFoam.Execution? In the later case one big difference is that NO server process is constructed automatically. What you could try is whether the --no-server-process-option makes a difference for the PyFoam.Applications.Runner-class

Anyway: If you're doing scripts PyFoam.Execution.UtilitiyRunner might be the better choice anyway (the PyFoam.Applications-classes are mainly for quickly "translating" what you did on the shell to a script)
Okay, that seems to be the thing: session killings stopped right after I added the --no-server-process option. Thank you!
leroyv is offline   Reply With Quote

Old   July 2, 2014, 05:29
Default
  #13
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by gschaider View Post
Hm. UtilityRunner from PyFoam.Applications or PyFoam.Execution? In the later case one big difference is that NO server process is constructed automatically. What you could try is whether the --no-server-process-option makes a difference for the PyFoam.Applications.Runner-class

Anyway: If you're doing scripts PyFoam.Execution.UtilitiyRunner might be the better choice anyway (the PyFoam.Applications-classes are mainly for quickly "translating" what you did on the shell to a script)
Hi,

I have been working on the problem again. I still get the getLinuxMem errors I mentioned. After testing on several configurations, it seems to be happening more frequently on my production configuration than on my dev computer. There are major differences between them, but the most important one is that the production computer has much faster processors than the development one.

This also happens only on very 'small' cases. When processing big amounts of data, I do not see these messages.

So my hypothesis would be: maybe this is just a matter of execution speed. Maybe the process started using any Runner class terminates so quickly that when any function using its PID is run, it cannot do its work as expected. What do you think about that?
leroyv is offline   Reply With Quote

Old   July 2, 2014, 12:37
Lightbulb I guess I was right
  #14
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
I added " && sleep N", where N is a number to be chosen for every task, to give PyFoam enough time to sort things out. So the typical call looks like
Code:
UtilityRunner(
    logname="PyFoamUtility.decomposePar",
    silent=not(verbose > 1),
    argv=[
        "decomposePar", 
        "-case", caseDirectory,
        "-force",
        " && sleep 1"
    ]).start()
Now, the nearly all of the getLinuxMem OSErrors are gone.
leroyv is offline   Reply With Quote

Old   July 2, 2014, 18:35
Default
  #15
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,920
Rep Power: 41
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by leroyv View Post
I added " && sleep N", where N is a number to be chosen for every task, to give PyFoam enough time to sort things out. So the typical call looks like
Code:
UtilityRunner(
    logname="PyFoamUtility.decomposePar",
    silent=not(verbose > 1),
    argv=[
        "decomposePar", 
        "-case", caseDirectory,
        "-force",
        " && sleep 1"
    ]).start()
Now, the nearly all of the getLinuxMem OSErrors are gone.
That is an OK workaround. But not really a solution as it slows things doen on machines where everything would have worked anyway.

The problem (as you said) is timing: pyFoam puts the utility into another thread so that it can "look at it from outside". Problem is that sometimes the utility is finished before PyFoam can have a first look.
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   July 3, 2014, 05:25
Default
  #16
Member
 
Vincent Leroy
Join Date: Jul 2012
Location: Rhode-Saint-Genèse, Belgium
Posts: 41
Rep Power: 5
leroyv is on a distinguished road
Quote:
Originally Posted by gschaider View Post
That is an OK workaround. But not really a solution as it slows things doen on machines where everything would have worked anyway.

The problem (as you said) is timing: pyFoam puts the utility into another thread so that it can "look at it from outside". Problem is that sometimes the utility is finished before PyFoam can have a first look.
Ok, now I understand things better. I guess in that case, since I don't do any log analysis, I should be running the utilities directly using the subprocess.call() function.
leroyv is offline   Reply With Quote

Reply

Tags
pyfoam

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
openfoam on 2 machines with passwordless ssh devayabir OpenFOAM 10 August 4, 2014 18:25
How to extract values with pyFoam? chriss85 OpenFOAM Running, Solving & CFD 13 January 27, 2014 21:16
pyFoam, pyFlu, extend: How does they relate? Horus OpenFOAM 9 May 21, 2011 12:49
pyFoam batch set up for OF 1.7 eelcovv OpenFOAM Running, Solving & CFD 3 January 19, 2011 09:38
Using SSH instead of RSH for parallel Eric CD-adapco 4 October 11, 2002 08:13


All times are GMT -4. The time now is 05:20.