HDD Storage Slow in Parallel Read/Write reconstructPar
 

All,

I searched the forum for something related to my issue but couldn't find much specific to my issue.

So i have a dual boot system and a shared HDD attached to the MOBO that is formatted to NTFS so that I can access data from windows as well as linux.

When i run cases on the HDD, namely parallel but also serial, I notice there is a large hang time when writing larger quantities of data, e.g. reconstructPar. I should say, more hang time than what I would expect, even from large case files with a high cell count.

When executing the reconstructPar command for larger cases 100K-1M or so cells, the utility will hang for anywhere from 10s to minutes before beginning to reconstructing the data. Even the transition from one time step to the next while reconstruction takes a good deal of time. Its so bad that I have restricted the number of parallel processors to use for the simulation because each additional processor drastically increases reconstruction time. When running these larger cases on my SSD, formatted to ext4, I dont see any excessive hang time (other than what would be expected when dealing with a large cell count and number of fields written).

The questions I have are:

1. Has anyone experienced this with an HDD?

2. Does anyone have any suggestion on what file system type I should use for a cross OS compatible storage drive such that linux (ubuntu 14.04) can handle the read/write operations required by openfoam?

3. should I abandon the requirement to have a cross OS compatible file system so that I can maximize performance when running large OF cases on the drive?

I will be purchasing a 2TB drive that has a higher spin speed and read/write speed. I just want to make sure I format everything properly before I make an investment.

Thanks all for the views and any feedback,

James

reconstructPar is pretty I/O intensive. I would try copying the data over from NTFS to a native Linux filesystem and see if performance improves. NTFS support in Linux has been focused on "making it work" versus "making it fast." You will likely have better results with ext3/4 or xfs filesystem serving up the data.

and if you need speed, buy a 128 or 256 GB SSD will make more differences than anything else. Use it as your storage for the current run and archive to the spin disk when you are done.

reconstructPar can be done in parallel as well. it reduces time immensly. i have the utility. in case you guys need it