Re: [netcdfgroup] nf90_char size

  • To: Dave Allured - NOAA Affiliate <dave.allured@xxxxxxxx>
  • Subject: Re: [netcdfgroup] nf90_char size
  • From: Wei-Keng Liao <wkliao@xxxxxxxxxxxxxxxx>
  • Date: Sat, 2 May 2020 22:48:34 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=northwestern.edu; dmarc=pass action=none header.from=northwestern.edu; dkim=pass header.d=northwestern.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VCIJN1qGUHJm6z1w630XJpAx3fQQdHJZzNsG1zrUQow=; b=RWQiTPPnYjokMFQvCDLptXwkFEc/2uwHybpYiUb5rMgazGp1l9QJVjrZ+AXC2FHbEIIVUWVyTefrXorBG+cuLcbdIIgMqufWneBa6wajzc4JFFpolN/f6G00aGL/agBoP3+d1FxTgAHgp2ZiTkCrdwMbvxod40tWSVNt2SbLPm8z8mzBkQPut9+AvfLnZycmdvzSvCb1T5cYjGVHESfPkhGE9SQaGvNxNowLiak2rOCm2sPAMqISA82fJoDghroMVH5NetZsinqI/zE4kin/VkmM6ydByqTZJ0aPo8eMnt49r6SsMhJfyW7xPiOEXKtScaXQ4o6buZFD1wETTYDALA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NJ9YhzWesoGhfAU/FZcQ1q/0heEW5bbPjDh/gDJRJPHVF0fk4x8AnbvFS1L7t8aWlY/AFyo1UUud8SRg9IXIXLDICKKmCbxWhCqF1vKNlyzN6hoZGeRx3I/H7bqUV5e8M0k7BdIXWsNJXRkYrObCyExQdh5OUJLe2FUN/G+OcVZZYMcaAGEYzp0TLhNqxo3PTDOfyjMdDpjlEetFu0zMN5qecV0hnC/7MUwepo2xQu9l3A2xemTNWA2rnew8R70eyQF57+o1EQ51BlFtj12Lvlw7N05tV8KTAc/5sk1KdW8/XtFkZDoWaYxYBDWGvPfhDuRyuK/lqX6kRmabc+4PQQ==
  • Authentication-results: noaa.gov; dkim=none (message not signed) header.d=none; noaa.gov; dmarc=none action=none header.from=northwestern.edu; 
Hi, Dave

Thanks for following up with the correct information about the dimension 
objects.
I admit that I am not familiar with the NetCDF4 dimension representation in 
HDF5.

Wei-keng

> On May 2, 2020, at 5:28 PM, Dave Allured - NOAA Affiliate 
> <dave.allured@xxxxxxxx> wrote:
> 
> Wei-king, thanks for the info on the latest release.  Minor detail, I found 
> that hidden dimension scales are still stored as arrays, but the arrays are 
> left unpopulated.  HDF5 stores these as sparse, which means no wasted space 
> in arrays that are never written.
> 
> For Davide, I concur with Wei-king that netcdf-C 4.7.4 is okay for your 
> purpose, and should not store wasted space.  Version 4.7.3 behaves the same 
> as 4.7.4.
> 
> I wonder when they changed that, some time between your 4.4.1.1 and 4.7.3.  
> Also you used HDF5 1.8.18, I used 1.10.5.  That should not make any 
> difference here, but perhaps it does.
> 
> 
> On Sat, May 2, 2020 at 1:01 PM Wei-Keng Liao <wkliao@xxxxxxxxxxxxxxxx> wrote:
> 
> If you used the latest NetCDF 4.7.4, the dimensions will be stored as scalars.
> 
> Wei-keng
> 
> > On May 2, 2020, at 1:42 PM, Davide Sangalli <davide.sangalli@xxxxxx> wrote:
> > 
> > Yeah, but  BS_K_linearized1 is just a dimension, how can it be 8 GB big ?
> > Same for  BS_K_linearized2, how can it be 3 GB big ?
> > These two are just two numbers
> > BS_K_linearized1 =  2,025,000,000
> > (it was chosen has a maximum variable size in my code to avoid overflowing 
> > the maximum allowed integer in standard precision)
> > BS_K_linearized2 = 781,887,360
> > 
> > D.
> > 
> > On 02/05/20 19:06, Wei-Keng Liao wrote:
> >> The dump information shows there are actually 8 datasets in the file.
> >> Below is the start offsets, sizes, and end offsets of individual datasets.
> >> There is not much padding space in between the datasets.
> >> According to this, your file is expected to be of size 16 GB.
> >> 
> >> dataset name                 start offset    size            end offset
> >> BS_K_linearized1             2,379           8,100,000,000   8,100,002,379
> >> BSE_RESONANT_COMPRESSED1_DONE        8,100,002,379   2,025,000,000   
> >> 10,125,002,379
> >> BSE_RESONANT_COMPRESSED2_DONE        10,125,006,475  2,025,000,000   
> >> 12,150,006,475
> >> BS_K_linearized2             12,150,006,475  3,127,549,440   15,277,555,915
> >> BSE_RESONANT_COMPRESSED3_DONE        15,277,557,963  781,887,360     
> >> 16,059,445,323
> >> complex                              16,059,447,371  8               
> >> 16,059,447,379
> >> BS_K_compressed1             16,059,447,379  99,107,168      16,158,554,547
> >> BSE_RESONANT_COMPRESSED1     16,158,554,547  198,214,336     16,356,768,883
> >> 
> >> Wei-keng
> >> 
> >>> On May 2, 2020, at 11:28 AM, Davide Sangalli <davide.sangalli@xxxxxx> 
> >>> wrote:
> >>> 
> >>> h5dump -Hp ndb.BS_COMPRESS0.005000_Q1
> 
> <snip>
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: