Thanks again.
I'll have a look to pnetcdf.
Another reason why we moved towards HDF5 was that, according to what I
know, they could be able to exploit different levels of memory
hierarchy in HPC simulations. Could pnetcdf do that as well ?
Besides that I'd really like some hints. Why could netcdf better than
HDF5, or viceversa. Please do your worst.
For the NF90_unlimited, we are already using it in time dependent
simulations in a way similar to the one you suggest. For the present
case instead I'm just filling a huge complex matrix. So the
interruption usually happens because there are limits on the
simulation time. I'd really need to check which elements were filled
and which were not without having any clue on the status.
Since you mentioned it. I'm very interested in the storage of sparse
matrices. My huge matrix is indeed quite sparse. How does that work ?
Best,
D
On Sun, May 3, 2020 at 12:48 AM +0200, "Wei-Keng Liao"
<wkliao@xxxxxxxxxxxxxxxx <mailto:wkliao@xxxxxxxxxxxxxxxx>> wrote:
Hi, Dave
Thanks for following up with the correct information about the dimension
objects.
I admit that I am not familiar with the NetCDF4 dimension representation in
HDF5.
Wei-keng
> On May 2, 2020, at 5:28 PM, Dave Allured - NOAA Affiliate wrote: > >
Wei-king, thanks for the info on the latest release.
Minor detail, I found that hidden dimension scales are still
stored as arrays, but the arrays are left unpopulated. HDF5 stores
these as sparse, which means no wasted space in arrays that are
never written. > > For Davide, I concur with Wei-king that
netcdf-C 4.7.4 is okay for your purpose, and should not store
wasted space. Version 4.7.3 behaves the same as 4.7.4. > > I
wonder when they changed that, some time between your 4.4.1.1 and
4.7.3. Also you used HDF5 1.8.18, I used 1.10.5. That should not
make any difference here, but perhaps it does. > > > On Sat, May
2, 2020 at 1:01 PM Wei-Keng Liao wrote: > > If you used the latest
NetCDF 4.7.4, the dimensions will be stored as scalars. > >
Wei-keng > > > On May 2, 2020, at 1:42 PM, Davide Sangalli wrote:
> > > > Yeah, but BS_K_linearized1 is just a dimension, how can it
be 8 GB big ? > > Same for BS_K_linearized2, how can it be 3 GB
big ? > > These two are just two numbers > > BS_K_linearized1 =
2,025,000,000 > > (it was chosen has a maximum variable size in my
code to avoid overflowing the maximum allowed integer in standard
precision) > > BS_K_linearized2 = 781,887,360 > > > > D. > > > >
On 02/05/20 19:06, Wei-Keng Liao wrote: > >> The dump information
shows there are actually 8 datasets in the file. > >> Below is the
start offsets, sizes, and end offsets of individual datasets. > >>
There is not much padding space in between the datasets. > >>
According to this, your file is expected to be of size 16 GB. > >>
> >> dataset name start offset size end offset > >>
BS_K_linearized1 2,379 8,100,000,000 8,100,002,379 > >>
BSE_RESONANT_COMPRESSED1_DONE 8,100,002,379 2,025,000,000
10,125,002,379 > >> BSE_RESONANT_COMPRESSED2_DONE 10,125,006,475
2,025,000,000 12,150,006,475 > >> BS_K_linearized2 12,150,006,475
3,127,549,440 15,277,555,915 > >> BSE_RESONANT_COMPRESSED3_DONE
15,277,557,963 781,887,360 16,059,445,323 > >> complex
16,059,447,371 8 16,059,447,379 > >> BS_K_compressed1
16,059,447,379 99,107,168 16,158,554,547 > >>
BSE_RESONANT_COMPRESSED1 16,158,554,547 198,214,336 16,356,768,883
> >> > >> Wei-keng > >> > >>> On May 2, 2020, at 11:28 AM, Davide
Sangalli wrote: > >>> > >>> h5dump -Hp ndb.BS_COMPRESS0.005000_Q1 > >
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
