[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [h5md-user] Strings again / parallel issue
From: |
Felix Höfling |
Subject: |
Re: [h5md-user] Strings again / parallel issue |
Date: |
Tue, 04 Feb 2014 17:01:42 +0100 |
User-agent: |
Opera Mail/12.16 (Linux) |
Am 04.02.2014, 16:05 Uhr, schrieb Pierre de Buyl
<address@hidden>:
Hi all,
I just discovered a serious annoyance with HDF5 variable-length types.
You
cannot use them in parallel [1].
This may have slipped through because nobody is doing parallel H5MD
right now,
however for large simulations in parallel (with existing code like
ESPResSo or
ESPResSo++) this will become a problem really fast.
I became interested in this because I was toying with topologies (not
yet part
of H5MD so no problem for now) but all the strings in H5MD are
variable-length
currently. I thus started to check the behaviour for our string data.
Unless someone has a solution to this problem (because I misunderstood
something), we probably need to change all VL strings to fixed-length
strings.
The only real annoyance is for arrays of strings (such as the box
boundary that
may hold strings of different lengths) were the maximum length must be
used for
the string type.
To put the timeframe into context, our paper [2] (me, peter and felix)
is accepted
now and we are waiting for the proofs. So this should be set within the
next few
days.
I attach parallel_string_attribute.f90 (fixed length attribute, works in
parallel) and parallel_vl_string_attribute.f90 (vl strings, doesn't work
as far
as I can tell). They require parallel hdf5, indeed.
Best,
Pierre
[1] http://www.hdfgroup.org/hdf5-quest.html#pvl
[2] http://arxiv.org/abs/1308.6382
I always felt uncomfortable with specialising to a certain String type. I
suggest to drop "VL" and allow _any_ kind of HDF5 String type for the
writer. The reader can easily handle the different cases by querying
H5Tis_variable_str
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-IsVariableString
For reading a string attribute, you may have a look at the code snippet
below.
Regards,
Felix
std::string read_string_attribute(hid_t attr_id)
{
std::string value;
hid_t type_id = H5Aget_type(attr_id));
htri_t is_varlen_str = H5Tis_variable_str(type_id));
if (is_varlen_str > 0) {
// read fixed-size string, allocate space in advance and let the
HDF5
// library take care about NULLTERM and NULLPAD strings
hsize_t size = H5Tget_size(type_id);
hid_t mem_type_id = H5Tcopy(H5T_C_S1);
H5Tset_size(mem_type_id, size);
value.resize(size, std::string::value_type());
H5Aread(attr_id, mem_type_id, &*value.begin());
} else {
// read variable-length string, memory will be allocated by HDF5 C
// library and must be freed by us
char *c_str;
if (H5Aread(attr_id, H5T_C_S1, &c_str) >= 0) {
value = c_str; // copy '\0'-terminated string
free(c_str);
}
}
H5Tclose(type_id);
H5Aclose(attr_id);
return value;
}
- Re: [h5md-user] Strings again / parallel issue, (continued)
- Re: [h5md-user] Strings again / parallel issue, Pierre de Buyl, 2014/02/05
- Re: [h5md-user] Strings again / parallel issue, Peter Colberg, 2014/02/05
- Re: [h5md-user] Strings again / parallel issue, Felix Höfling, 2014/02/06
- Re: [h5md-user] Strings again / parallel issue, Peter Colberg, 2014/02/06
- Re: [h5md-user] Strings again / parallel issue, Pierre de Buyl, 2014/02/06
- Re: [h5md-user] Strings again / parallel issue, Felix Höfling, 2014/02/07
- Re: [h5md-user] Strings again / parallel issue, Pierre de Buyl, 2014/02/07
- Re: [h5md-user] Strings again / parallel issue, Peter Colberg, 2014/02/07
- Re: [h5md-user] Strings again / parallel issue, Pierre de Buyl, 2014/02/06
- Re: [h5md-user] Strings again / parallel issue, Peter Colberg, 2014/02/06
Re: [h5md-user] Strings again / parallel issue,
Felix Höfling <=
Re: [h5md-user] Strings again / parallel issue, Felix Höfling, 2014/02/04