CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Paraview & paraFoam

Big-endian to little-endian conversion. How to do it in OpenFOAM?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 2 Post By opedrofunk

Reply
 
LinkBack Thread Tools Display Modes
Old   January 28, 2015, 06:54
Default Big-endian to little-endian conversion. How to do it in OpenFOAM?
  #1
New Member
 
Matteo Cerminara
Join Date: Feb 2012
Posts: 15
Rep Power: 6
Pagoda is on a distinguished road
Hi all,
I am trying to open some data saved in binary in an infrastructure adopting the big-endian protocol in an infrastructure that is using the little-endian protocol.

How could I make the standard .foam paraview reader change the byte order of the files in the constant/polymesh and temporal directories?

Alternatively, I tried to use foamToVTK, but when the mesh is written in binary I obtain a segmentation fault error.

Does anybody have some idea to face this compatibility problem?

Thank you very much!

Matteo
Pagoda is offline   Reply With Quote

Old   January 31, 2015, 10:51
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Greetings Matteo,

That's one very tricky situation you're dealing with.
AFAIK, OpenFOAM itself does not have any feature that converts binary data between little and big-endian. Nonetheless, it might be possible to at least code this into foamToVTK, for writing the VTK files in binary in a specific endian.

Any chance you can specify what combination you have with OpenFOAM vs ParaView and big-endian vs little-endian?

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   February 5, 2015, 11:37
Default
  #3
New Member
 
Matteo Cerminara
Join Date: Feb 2012
Posts: 15
Rep Power: 6
Pagoda is on a distinguished road
Greetings Bruno,
I would like to post-process data computed by a Big-endian machine in a machine which is Little-endian.
I know this by typing
Code:
echo -n I | od -to2 | head -n1
in a bash terminal.
The output of this command has 0 as last number in the Big-endian machine while it has 1 in a Little-endian machine.

I am using OpenFOAM 2.1.1 and paraview 4.2.0.

Yes, I would like to have the utility foamToVTK capable to convert from one binary format to another.
Thank you for your reply!

Best regards,
Matteo
Pagoda is offline   Reply With Quote

Old   February 7, 2015, 07:38
Default
  #4
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Hi Matteo,

The simplest solution would be for you to run foamToVTK with the option "-ascii" on the machine where the case was executed originally.

After a bit of thinking, there are two ways to address this issue:
  1. Have an option in foamToVTK which flips the byte order when saving to VTK in binary format. This means that the utility must be executed on the original machine.
  2. Have a utility in OpenFOAM which reinterprets dictionaries, without actually loading them as field data. In other words, all files would be read as dictionary files and the fields would be read as arrays of bytes, then their bytes would be flipped according to the bit count.
The first way is the easiest one to do, I think. I'll try to look into this during this current weekend.

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   February 7, 2015, 08:43
Default
  #5
New Member
 
Matteo Cerminara
Join Date: Feb 2012
Posts: 15
Rep Power: 6
Pagoda is on a distinguished road
Hi Bruno,
thank you very much for your help!

Even if the first option is more easy, as also using foamFormatConvert and selecting the ascii format in controlDict, it would be desirable to be able to analyze OpenFOAM fields in every machine.

In particular, I have a big simulation processed in an HPC infrastructure (the Big-endian one), saved in a local storage system which is Little-endian. Moving all the data back to the HPC infrastructure is very time expensive and, anyway, save all the data in ascii format is not very good from the point of view of economy of data storage (even if you compress them).

I do not know if possible, but the best would be to have paraview .foam and .vtk readers able to recognize the format of the binary (as for example the XML file format can do).

I know that in paraview vtk reader exist the following commands:
Code:
SetDataByteOrderToBigEndian()
SetDataByteOrderToLittleEndian()
but I have no idea on how to use them.

Alternatively, as you proposed, I think an utility capable to invert the order of the binary fields would be very useful.


Best regards,
Matteo

------------

Maybe this document can be useful

vtk file formats

Last edited by wyldckat; February 7, 2015 at 09:14. Reason: posted few minutes apart
Pagoda is offline   Reply With Quote

Old   February 7, 2015, 09:10
Default
  #6
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Hi Matteo,

Well, OpenFOAM's source code for foamToVTK already has endianess handling in the file "writeFuns.C".

In addition, according to that document you provided in the previous post, the final VTK file shouldn't have any problems if you can export it on the original Big Endian system.

Please do the following steps, so that I can debug the resulting VTK file without needing access to a "big endian" machine:
  1. Go to a folder in the "big endian" machine where you can create a new folder.
  2. Run:
    Code:
    cp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity cavityBE
    cd cavityBE
    blockMesh
    icoFoam
    foamToVTK
    gzip VTK/cavityBE_100.vtk
  3. Attach the file "VTK/cavityBE_100.vtk.gz" to your next post.
  4. In addition, unpack this file on your "little endian" machine:
    Code:
    gunzip cavityBE_100.vtk.gz
    And open the file "cavityBE_100.vtk" in ParaView.
Best regards,
Bruno

----------------

edit:
I only saw that you had made 2 posts and not just the last one. I've merged the two.
OK, then here's a question, just to confirm: Are you trying to open with the ".foam" or the ".OpenFOAM" file extension in ParaView? Because the solutions for each specific format are very different.

Last edited by wyldckat; February 7, 2015 at 09:17. Reason: see "edit:"
wyldckat is offline   Reply With Quote

Old   February 7, 2015, 12:47
Default
  #7
New Member
 
Matteo Cerminara
Join Date: Feb 2012
Posts: 15
Rep Power: 6
Pagoda is on a distinguished road
Hi Bruno,
I enclosed here the vtk binary file in Big-endian format.

If I try to open it with paraview in a Little endian machine I get an error causing paraview to close.

The extension I use for reading OpenFoam files is ".foam", because it allows me to post process decomposed cases.

Best regards,
Matteo
Attached Files
File Type: gz cavityBE_100.vtk.gz (18.2 KB, 2 views)
Pagoda is offline   Reply With Quote

Old   February 7, 2015, 13:31
Default
  #8
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Hi Matteo,

Thanks for the VTK file!

Then if it's for ".foam", you can start doing some "homework" by compiling the latest vtkPOFFReader for ParaView on the machine with "little endian": https://github.com/7islands/vtkPOFFReader

Instructions are also on the wiki: http://openfoamwiki.net/index.php/Co...r_for_ParaView

In the mean time, I'll start by seeing what can be done on foamToVTK and/or for the VTK reader in ParaView.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   February 7, 2015, 13:52
Default
  #9
Member
 
Peter
Join Date: Feb 2015
Location: California
Posts: 59
Rep Power: 3
opedrofunk is on a distinguished road
Another solution, depending on your requirements, might be to simply post-process with paraview on the HPC directly. This is what paraview was designed to do. And it's the way I typically work - especially with large datasets - and allows you to leverage the power of your HPC center for post-processing as well.

The way to do it:

1) Open paraview

2) Go to File -> Connect (or click the "Connect" icon)

3) Click "Add Server" and enter a name (anything you want). Leave it as "Client/Server". Enter "localhost" in the host dialog, and 11111 in the port dialog (you can enter any open port number here). Then click "Configure"

4) Select "Command" from the drop down (a text area will appear where you can enter the command to be executed when connecting to your remote HPC server). Set "wait" to 10 seconds. The way paraview was designed to work is via ssh tunneling. This is the command I use (with name and server made generic):

Code:
ssh -g -L 11111:localhost:11542 username@remoteserveraddress "pvserver --use-offscreen-rendering -sp=11542"
It runs ssh and binds port 11111 (or whatever you specified) on your machine (localhost) to an open port on the remote HPC machine (11542, in this case). It also starts a pvserver on the HPC with the "--use-offscreen-rendering" option and the "-sp" (server port) option set to 11542 (or which ever open port on the remote machine you want to use).

You will of course need to use passwordless login to the machine (i.e., run ssh-copy-id once to copy your public key to the HPC)

Thats it!

Then you can connect, and use paraview the normal way (only you'll be driving an HPC on the back end). And paraview saves your configuration so you only have to do this once. Then you can reconnect with one click.

To run paraview in parallel on the HPC, just change the command to something like:

Code:
ssh -g -L 11111:localhost:11542 username@remoteserveraddress "mpirun -np 16 pvserver --use-offscreen-rendering -sp=11542"
This also allows you to let long-running commands (like, creating a high-res animation for a large dataset, for example) run in the background on the HPC. i.e. you can now use the "Disconnect before saving animation" option.

Notes: Depending on the way the HPC is configured, you might need to provide the full path to pvserver. i.e. /path/to/pvserver. Also, you must use the same version of paraview on your local machine as is installed on the remote machine.

Hope this helps!
Peter
wyldckat and Pagoda like this.
opedrofunk is offline   Reply With Quote

Old   February 7, 2015, 15:23
Default
  #10
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Greetings to all!

@Peter: Many thanks for the detailed instructions!

@Matteo: Try this out: https://github.com/wyldckat/foamToVTKSwapEndian - instructions for downloading, building and using are provided on that page. Keep in mind that the build utility must be used in the original machine, since it needs to load the original fields.

As for the ability to open ".foam" files directly on ParaView with an endianess option, I also need the full test case "cavityBE" you created before on the Big Endian machine, so that I can test it. In addition, let me know if you have issues building the "vtkPOFFReader" add-on for ParaView.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   February 8, 2015, 07:09
Default
  #11
New Member
 
Matteo Cerminara
Join Date: Feb 2012
Posts: 15
Rep Power: 6
Pagoda is on a distinguished road
Hi guys, very nice informations!

Peter, thank you very much for your instructions! Unfortunately The HPC machine I am working with is just for calculations, no visual nodes are provided and paraview is not installed there.

Bruno, I will try as soon as possible your suggestions with paraview, in the meanwhile I enclosed here the cavityBE case. I removed time 0.1 in there in order to not exceed the forum file size limit.


All the best,
Matteo
Attached Files
File Type: gz cavityBE.tar.gz (87.3 KB, 1 views)
Pagoda is offline   Reply With Quote

Old   February 11, 2015, 19:36
Default
  #12
Member
 
Peter
Join Date: Feb 2015
Location: California
Posts: 59
Rep Power: 3
opedrofunk is on a distinguished road
Hi,
You might try downloading the linux binary (here) and running it from your home directory on the HPC. In that case, give the full path (i.e., /path/to/home/directory/and/pvserver ) when you start pvserver.

Anyway, might be worth a shot if it can save you from having to copy loads of data.

Good luck!
Peter
opedrofunk is offline   Reply With Quote

Old   April 4, 2015, 15:46
Default
  #13
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,743
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Greetings to all!

@Matteo: I've finally managed to look into this. Any chance you can test the updated plug-in code: https://github.com/wyldckat/vtkPOFFR...e/EndianOption
You can download the source code as a ZIP file by clicking on the button (to the right) named "Download ZIP"

As for building it, it's done as explained here: http://openfoamwiki.net/index.php/Co...ng_the_plugins

If you need help with building it from source code, please let me know.

The new option is shown on this image:


It's the "Swap Endianness of the binary data".

----------
Edit: Forgot to mention that you can then open the case by using:
Code:
paraFoam -builtin
And I've submitted a pull request for this new feature as well: https://github.com/7islands/vtkPOFFReader/pull/9
----------

Best regards,
Bruno
Attached Images
File Type: png New Option.png (10.6 KB, 19 views)

Last edited by wyldckat; April 4, 2015 at 15:58. Reason: see "Edit:"
wyldckat is offline   Reply With Quote

Reply

Tags
big-endian, little-endian, openfoam, paraview

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Mesh Conversion to Openfoam problem Ahadi OpenFOAM Meshing & Mesh Conversion 0 June 13, 2014 09:28
Frequently Asked Questions about Installing OpenFOAM wyldckat OpenFOAM Installation 0 January 1, 2014 20:21
OpenFOAM Foundation releases OpenFOAM 2.2.2 opencfd OpenFOAM Announcements from ESI-OpenCFD 0 October 14, 2013 07:18
Map data from a big to a small domain in OpenFOAM sagnikmazumdar OpenFOAM 1 January 24, 2013 04:12
OpenFOAM Debian packaging current status problems and TODOs oseen OpenFOAM Installation 9 August 26, 2007 13:50


All times are GMT -4. The time now is 18:19.