Thanks Dennis, I am happy with your proposal.
Cheers,
Joaquín
________________________________________
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
<netcdfgroup-bounces@xxxxxxxxxxxxxxxx> on behalf of Dennis Heimbigner
<dmh@xxxxxxxx>
Sent: Sunday, February 25, 2018 7:46 PM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] How to uncompress with nccopy in netCDF-C 4.6.0
This was an oversight on my part. The old behavior should apply
and I will revert it to not use a filter as a default.
However, it occurs to me that it would be desirable to have
a shorthand notation to tell nccopy to use the same filter
from the input. There are probably two cases.
1. transfer the filter from input to output for all variables
2. transfer the filter from input to output for specified variables.
Tentatively I propose the following option flags:
1. -F all => transfer filter for all variables
2. -F <var>: => transfer filter for <var>
#2 could be repeated for different variables.
Comments?
=Dennis Heimbigner
Unidata
On 2/25/2018 5:06 AM, Joaquin Rodriguez Guerra wrote:
>
> Hi there,
>
>
> With netCDF-C now supporting HDF dynamic filters, the behaviour of
> nccopy has slightly changed. In the past nccopy could be used to
> uncompress files previously compressed with custom HDF dynamic
> filters. Nccopy used to read <infile>, relying on HDF to invoke the
> decompression using the filter, and then it would create a new netCDF
> file, <outfile>, uncompressed. In version 4.6.0, nccopy also reads
> <infile> relying on HDF as before, but then writes the new <outfile>
> applying the filter options defined in <infile>. I guess this is the
> new default behaviour when not passing the -F argument.
>
> Is there any option that can be passed in the -F argument to tell
> nccopy not to apply any compression algorithm?
>
> Thanks! and keep up the good work! Dynamic filter write support is a
> very hice feature!
>
> Here an example, with a sample.nc file previously compressed with a
> custom HDF filter.
>
> jrgu@linux-ozpe:~/FiltersTest/Test> ncdump -h -s sample.nc
> netcdf sample {
> dimensions:
> number_of_columns = 30 ;
> number_of_rows = 60 ;
> variables:
> short effective_radiance(number_of_rows, number_of_columns) ;
> effective_radiance:_Storage = "chunked" ;
> effective_radiance:_ChunkSizes = 60, 30 ;
> effective_radiance:_Endianness = "little" ;
> effective_radiance:_Filter = "56782,2,1,60,30,16,1,0,0,0,0,0,0,0" ; //
> filterID + filter options (cd values)
> ubyte pixel_quality(number_of_rows, number_of_columns) ;
> pixel_quality:_Storage = "chunked" ;
> pixel_quality:_ChunkSizes = 60, 30 ;
> pixel_quality:_Filter = "56782,1,1,60,30,8,1,0,0,0,0,0,0,0" ; //
> filterID + filter options (cd values)
>
> // global attributes:
> :_SuperblockVersion = 2 ;
> :_IsNetcdf4 = 1 ;
> :_Format = "netCDF-4" ;
> }
>
> I have added some debugging messages to the custom HDF dynamic filter
> and obtained the following when using nccopy:
>
> * Nccopy in NetCDF-c 4.4 and 4.5: the decoding functionality in the
> filter is invoked once for each variable when reading in_file:
>
> jrgu@linux-ozpe:~/FiltersTest/Test>
> ../netcdf-c-4.5.0/ncdump/nccopy sample.nc sample_nccopy45.nc
>
> /Deconding/
> /CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ... /
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 2/
> /Decompression successfull/
> /Decode End: 3600/
>
> /Deconding/
> /CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, ... /
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 1/
> /Decompression successfull/
> /Decode End: 1800/
>
>
> * //Nccopy in NetCDF-c 4.6: the decoding functionality in the filter
> is invoked once for each variable when reading in_file. The encoding
> functionality in the filter is invoked once for each variable when
> writing out_file
>
> jrgu@linux-ozpe:~/FiltersTest/Test> nccopy sample.nc
> sample_nccopy46.nc
> /Deconding/
> /CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ... /
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 2/
> /Decompression successfull/
> /Decode End: 3600/
>
> /Deconding/
> /CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, .../
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 1/
> /Decompression successfull/
> /Decode End: 1800/
>
> /Encoding/
> /CD VALUES 2, 1, 60, 30, 2, 1, 60, 30, 16, 1, .../
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 2/
> /Pixel: 1800/
> /ERROR: Error in jpeglsCompress: Parameter values are not
> supported by CharLS./
> /Compression failed/
> /ERROR: Error during the JPEG-LS compression of the dataset./
> /Encode End -> Result: 0/
>
> /Encoding/
> /CD VALUES 1, 1, 60, 30, 1, 1, 60, 30, 8, 1 /
> /Components: 1, Lines: 60, Samples: 30/
> /DataBytes: 1/
> /Pixel: 1800/
> /ERROR: Error in jpeglsCompress: Parameter values are not
> supported by CharLS./
> /Compression failed/
> /ERROR: Error during the JPEG-LS compression of the dataset./
> /Encode End -> Result: 0/
>
>
> Cheers,
> Joaquín
>
>
>
> Any email message from EUMETSAT is sent in good faith but shall
> neither be binding nor construed as constituting a commitment by
> EUMETSAT, except where provided for in a written agreement or contract
> or if explicitly stated in the email. Please note that any views or
> opinions presented in this email are solely those of the sender and do
> not necessarily represent those of EUMETSAT. This message and any
> attachments are intended for the sole use of the addressee(s) and may
> contain confidential and privileged information. Any unauthorised use,
> disclosure, dissemination or distribution (in whole or in part) of its
> contents is not permitted. If you received this message in error,
> please notify the sender and delete it from your system.
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
Any email message from EUMETSAT is sent in good faith but shall neither be
binding nor construed as constituting a commitment by EUMETSAT, except where
provided for in a written agreement or contract or if explicitly stated in the
email. Please note that any views or opinions presented in this email are
solely those of the sender and do not necessarily represent those of EUMETSAT.
This message and any attachments are intended for the sole use of the
addressee(s) and may contain confidential and privileged information. Any
unauthorised use, disclosure, dissemination or distribution (in whole or in
part) of its contents is not permitted. If you received this message in error,
please notify the sender and delete it from your system.
