Big-endian to little-endian conversion. How to do it in OpenFOAM?
 

Hi all,
I am trying to open some data saved in binary in an infrastructure adopting the big-endian protocol in an infrastructure that is using the little-endian protocol.

How could I make the standard .foam paraview reader change the byte order of the files in the constant/polymesh and temporal directories?

Alternatively, I tried to use foamToVTK, but when the mesh is written in binary I obtain a segmentation fault error.

Does anybody have some idea to face this compatibility problem?

Thank you very much!

Matteo

Greetings Matteo,

That's one very tricky situation you're dealing with.
AFAIK, OpenFOAM itself does not have any feature that converts binary data between little and big-endian. Nonetheless, it might be possible to at least code this into foamToVTK, for writing the VTK files in binary in a specific endian.

Any chance you can specify what combination you have with OpenFOAM vs ParaView and big-endian vs little-endian?

Best regards,
Bruno

Greetings Bruno,
I would like to post-process data computed by a Big-endian machine in a machine which is Little-endian.
I know this by typing
in a bash terminal.
The output of this command has 0 as last number in the Big-endian machine while it has 1 in a Little-endian machine.

I am using OpenFOAM 2.1.1 and paraview 4.2.0.

Yes, I would like to have the utility foamToVTK capable to convert from one binary format to another.
Thank you for your reply!

Best regards,
Matteo

Hi Matteo,

The simplest solution would be for you to run foamToVTK with the option "-ascii" on the machine where the case was executed originally.

After a bit of thinking, there are two ways to address this issue:

1. Have an option in foamToVTK which flips the byte order when saving to VTK in binary format. This means that the utility must be executed on the original machine.
2. Have a utility in OpenFOAM which reinterprets dictionaries, without actually loading them as field data. In other words, all files would be read as dictionary files and the fields would be read as arrays of bytes, then their bytes would be flipped according to the bit count.

The first way is the easiest one to do, I think. I'll try to look into this during this current weekend.

Best regards,
Bruno

Hi Bruno,
thank you very much for your help!

Even if the first option is more easy, as also using foamFormatConvert and selecting the ascii format in controlDict, it would be desirable to be able to analyze OpenFOAM fields in every machine.

In particular, I have a big simulation processed in an HPC infrastructure (the Big-endian one), saved in a local storage system which is Little-endian. Moving all the data back to the HPC infrastructure is very time expensive and, anyway, save all the data in ascii format is not very good from the point of view of economy of data storage (even if you compress them).

I do not know if possible, but the best would be to have paraview .foam and .vtk readers able to recognize the format of the binary (as for example the XML file format can do).

I know that in paraview vtk reader exist the following commands:
but I have no idea on how to use them.

Alternatively, as you proposed, I think an utility capable to invert the order of the binary fields would be very useful.


Best regards,
Matteo

------------

Maybe this document can be useful

vtk file formats

Hi Matteo,

Well, OpenFOAM's source code for foamToVTK already has endianess handling in the file "writeFuns.C".

In addition, according to that document you provided in the previous post, the final VTK file shouldn't have any problems if you can export it on the original Big Endian system.

Please do the following steps, so that I can debug the resulting VTK file without needing access to a "big endian" machine:

1. Go to a folder in the "big endian" machine where you can create a new folder.
2. Run:
   
   Code:

   ---

   cp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity cavityBE
cd cavityBE
blockMesh
icoFoam
foamToVTK
gzip VTK/cavityBE_100.vtk

   ---

3. Attach the file "VTK/cavityBE_100.vtk.gz" to your next post.
4. In addition, unpack this file on your "little endian" machine:
   
   Code:

   ---

   gunzip cavityBE_100.vtk.gz

   ---

   And open the file "cavityBE_100.vtk" in ParaView.

Best regards,
Bruno

----------------

edit:
I only saw that you had made 2 posts and not just the last one. I've merged the two.
OK, then here's a question, just to confirm: Are you trying to open with the ".foam" or the ".OpenFOAM" file extension in ParaView? Because the solutions for each specific format are very different.

Hi Bruno,
I enclosed here the vtk binary file in Big-endian format.

If I try to open it with paraview in a Little endian machine I get an error causing paraview to close.

The extension I use for reading OpenFoam files is ".foam", because it allows me to post process decomposed cases.

Best regards,
Matteo

Hi Matteo,

Thanks for the VTK file!

Then if it's for ".foam", you can start doing some "homework" by compiling the latest vtkPOFFReader for ParaView on the machine with "little endian": https://github.com/7islands/vtkPOFFReader

Instructions are also on the wiki: http://openfoamwiki.net/index.php/Co...r_for_ParaView

In the mean time, I'll start by seeing what can be done on foamToVTK and/or for the VTK reader in ParaView.

Best regards,
Bruno

Another solution, depending on your requirements, might be to simply post-process with paraview on the HPC directly. This is what paraview was designed to do. And it's the way I typically work - especially with large datasets - and allows you to leverage the power of your HPC center for post-processing as well.

The way to do it:

1) Open paraview

2) Go to File -> Connect (or click the "Connect" icon)

3) Click "Add Server" and enter a name (anything you want). Leave it as "Client/Server". Enter "localhost" in the host dialog, and 11111 in the port dialog (you can enter any open port number here). Then click "Configure"

4) Select "Command" from the drop down (a text area will appear where you can enter the command to be executed when connecting to your remote HPC server). Set "wait" to 10 seconds. The way paraview was designed to work is via ssh tunneling. This is the command I use (with name and server made generic):

It runs ssh and binds port 11111 (or whatever you specified) on your machine (localhost) to an open port on the remote HPC machine (11542, in this case). It also starts a pvserver on the HPC with the "--use-offscreen-rendering" option and the "-sp" (server port) option set to 11542 (or which ever open port on the remote machine you want to use).

You will of course need to use passwordless login to the machine (i.e., run ssh-copy-id once to copy your public key to the HPC)

Thats it!

Then you can connect, and use paraview the normal way (only you'll be driving an HPC on the back end). And paraview saves your configuration so you only have to do this once. Then you can reconnect with one click.

To run paraview in parallel on the HPC, just change the command to something like:

This also allows you to let long-running commands (like, creating a high-res animation for a large dataset, for example) run in the background on the HPC. i.e. you can now use the "Disconnect before saving animation" option.

Notes: Depending on the way the HPC is configured, you might need to provide the full path to pvserver. i.e. /path/to/pvserver. Also, you must use the same version of paraview on your local machine as is installed on the remote machine.

Hope this helps!
Peter

Greetings to all!

@Peter: Many thanks for the detailed instructions!

@Matteo: Try this out: https://github.com/wyldckat/foamToVTKSwapEndian - instructions for downloading, building and using are provided on that page. Keep in mind that the build utility must be used in the original machine, since it needs to load the original fields.

As for the ability to open ".foam" files directly on ParaView with an endianess option, I also need the full test case "cavityBE" you created before on the Big Endian machine, so that I can test it. In addition, let me know if you have issues building the "vtkPOFFReader" add-on for ParaView.

Best regards,
Bruno

Hi guys, very nice informations!

Peter, thank you very much for your instructions! Unfortunately The HPC machine I am working with is just for calculations, no visual nodes are provided and paraview is not installed there.

Bruno, I will try as soon as possible your suggestions with paraview, in the meanwhile I enclosed here the cavityBE case. I removed time 0.1 in there in order to not exceed the forum file size limit.


All the best,
Matteo

Hi,
You might try downloading the linux binary (here) and running it from your home directory on the HPC. In that case, give the full path (i.e., /path/to/home/directory/and/pvserver ) when you start pvserver.

Anyway, might be worth a shot if it can save you from having to copy loads of data.

Good luck!
Peter

Greetings to all!

@Matteo: I've finally managed to look into this. Any chance you can test the updated plug-in code: https://github.com/wyldckat/vtkPOFFR...e/EndianOption
You can download the source code as a ZIP file by clicking on the button (to the right) named "Download ZIP" ;)

As for building it, it's done as explained here: http://openfoamwiki.net/index.php/Co...ng_the_plugins

If you need help with building it from source code, please let me know.

The new option is shown on this image:
http://www.cfd-online.com/Forums/att...chmentid=38329

It's the "Swap Endianness of the binary data".

----------
Edit: Forgot to mention that you can then open the case by using:
And I've submitted a pull request for this new feature as well: https://github.com/7islands/vtkPOFFReader/pull/9
----------

Best regards,
Bruno