A cluster has lots of unused distributed disk space on the nodes. Has anybody used PVFS2 to use that space for storage? I'm looking for advice and warnings of pitfalls.

Ideally, when you perform computations on a cluster the data should be distributed as well (each node should dump data locally). So, if there is "waste" perhaps the program logic needs change first!

Adrin Gharakhani

Well, that's not particularly relevant when the program runs fully in memory, then each node does not need to dump any data locally. But each cluster node has a fair amount of spare disk space, which would be a nice place to store output files.

I don't know the type of problems you're running, and the objective of the simulations, but data is dumped/stored regardless... If you simulate unsteady flow problems and want to examine the time evolution of various parameters, you end up requiring giga-terabytes of disk space (even when the program itself is small and runs in memory), in which case there is not only no wasted disk space but demand for extra disks... Anyway, I digress from your original question

Adrin Gharakhani

Parallel file systems generally work very well when you have 1 or more dedicated i/o servers to mount a big disk farm on many compute nodes. Doing what you want (combining disks from all over the place) will work but its very inefficient. Every compute node will also be tapped as an I/O server as well with the result that your calculation will be interrupted randomly on different nodes and your scaling will fall off. It doesn't do I/O well and it slows your calculation. Everyone I know who has tried it (with other systems, not PVFS) have given up.