Re: [h5md-user] Dataset layouts

[Peter Colberg]

On Tue, Jun 16, 2015 at 11:30:28AM +0200, Felix Höfling wrote:
> Peter, did I understand correctly: parallel reading of a dataset must take
> into account whether the dataset is compact or not, otherwise the data are
> inconsistent between the MPI processes? (Actually, such a mis-use of the
> HDF5 library should raise an exception in my opinion.)

Fortunately not, the inconsistency only arises when only one process
writes to the compact dataset, and subsequently all processes read
from that dataset. Metadata writes go to the per-process cache, and
metadata reads are from the per-process cache. To keep the per-process
caches in sync, metadata writes must be collective.

> With respect to writing, I don't see any need to require compactness. The
> writer application "knows" whether it uses the MPI interface or not and can
> act accordingly. Second, h5py does a great job in writing H5MD files so far.
> I would not like to break this kind of support by making compactness
> mandatory.

I agree, though h5py should allow the compact layout for efficiency.

> For reading, on the other hand, the implementation of a reader has to be
> simple for the sake of robustness. Querying the storage layout before
> reading may be one complication that can be avoided by specifying the
> layout. (This reminds me of an endless discussion about the string type.)

I tested what happens when all processes read a scalar dataset with
contiguous layout. It actually works fine. I get the same read times
as for the compact layout.

How about we include in the specification that the scalar "step"/"time"
dataset SHOULD use a compact layout?

Peter

hdf5_contiguous_dset.c
Description: Text Data

hdf5_compact_dset.c
Description: Text Data