Thanks for finding this and letting us know!
I have elevated this to an issue on GitHub so it gets proper attention:
https://github.com/Unidata/netcdf-c/issues/1436
Further discussion would be most helpful if it were in the issue, where we
can easily track it.
Keep on netCDFing!
Ed Hartnett
On Thu, Jul 11, 2019 at 7:51 AM Mike Chen <mike.scchen@xxxxxxxxx> wrote:
> Hi,
> We have a storage shortage and we'd like to leverage the compression to
> save space.
> The idea works fine for some time.
> However, I was testing different NetCDF versions and happens to find that
> the nccopy to compressed NetCDF-4 file is magnitudes slower on version
> 4.6.0+.
>
> The test method follows:
> OS: CentOS 6.10
> HDF5: 1.10.5
> NetCDF: 4.4.1.1, 4.5.0, 4.6.2, 4.7.0, HPC-NetCDF 4.6.2.4 from NOAA
> All building uses system-provided GCC suite 4.4.1.
>
> The test is simple: nccopy a ~400MB classic NetCDF file to NetCDF-4 with:
> nccopy -4 -d 1 -s input.nc output.nc4
> With version v4.5.0 and lower, the process takes ~7 seconds.
> HPC-NetCDF also gives similar speed.
> (gzip-ing the same file takes similar time.)
> But with version v4.6.2+, the copy can take ~25 minutes!
>
> Our production system still uses 4.4.0, so the speed should be okay.
> But I'd like to know more so I can plan the updates accordingly.
>
> Mike Chen
> Research Assistant / System Engineer
> Dept. of Atmospheric Science
> National Taiwan University
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
