The simplest rule for nccopy would be to preserve all characteristics of
the input file, except when to instructed to change something. If you like
this rule, then the old behavior of automatically de-filtering should be
left behind. That previous behavior was a nice convenience feature before
filter access was available in netcdf 4.6.0.
So my suggestion is to keep the new behavior, preserving filters by
default. Then add a shorthand option to remove all filters, something like
-F none.
--Dave
On Sun, Feb 25, 2018 at 11:46 AM, Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> This was an oversight on my part. The old behavior should apply
> and I will revert it to not use a filter as a default.
> However, it occurs to me that it would be desirable to have
> a shorthand notation to tell nccopy to use the same filter
> from the input. There are probably two cases.
> 1. transfer the filter from input to output for all variables
> 2. transfer the filter from input to output for specified variables.
>
> Tentatively I propose the following option flags:
> 1. -F all => transfer filter for all variables
> 2. -F <var>: => transfer filter for <var>
> #2 could be repeated for different variables.
>
> Comments?
>
> =Dennis Heimbigner
> Unidata
>
>
> On 2/25/2018 5:06 AM, Joaquin Rodriguez Guerra wrote:
>
>>
>> Hi there,
>>
>>
>> With netCDF-C now supporting HDF dynamic filters, the behaviour of nccopy
>> has slightly changed. In the past nccopy could be used to uncompress files
>> previously compressed with custom HDF dynamic filters. Nccopy used to read
>> <infile>, relying on HDF to invoke the decompression using the filter, and
>> then it would create a new netCDF file, <outfile>, uncompressed. In version
>> 4.6.0, nccopy also reads <infile> relying on HDF as before, but then writes
>> the new <outfile> applying the filter options defined in <infile>. I guess
>> this is the new default behaviour when not passing the -F argument.
>>
>> Is there any option that can be passed in the -F argument to tell nccopy
>> not to apply any compression algorithm?
>>
>> Thanks! and keep up the good work! Dynamic filter write support is a very
>> hice feature!
>>
>> Here an example, with a sample.nc file previously compressed with a
>> custom HDF filter.
>>
>> jrgu@linux-ozpe:~/FiltersTest/Test> ncdump -h -s sample.nc
>> netcdf sample {
>> dimensions:
>> number_of_columns = 30 ;
>> number_of_rows = 60 ;
>> variables:
>> short effective_radiance(number_of_rows, number_of_columns) ;
>> effective_radiance:_Storage = "chunked" ;
>> effective_radiance:_ChunkSizes = 60, 30 ;
>> effective_radiance:_Endianness = "little" ;
>> effective_radiance:_Filter = "56782,2,1,60,30,16,1,0,0,0,0,0,0,0" ; //
>> filterID + filter options (cd values)
>> ubyte pixel_quality(number_of_rows, number_of_columns) ;
>> pixel_quality:_Storage = "chunked" ;
>> pixel_quality:_ChunkSizes = 60, 30 ;
>> pixel_quality:_Filter = "56782,1,1,60,30,8,1,0,0,0,0,0,0,0" ; //
>> filterID + filter options (cd values)
>>
>> // global attributes:
>> :_SuperblockVersion = 2 ;
>> :_IsNetcdf4 = 1 ;
>> :_Format = "netCDF-4" ;
>> }
>>
>> I have added some debugging messages to the custom HDF dynamic filter and
>> obtained the following when using nccopy:
>>
>> * Nccopy in NetCDF-c 4.4 and 4.5: the decoding functionality in the
>> filter is invoked once for each variable when reading in_file:
>>
>> jrgu@linux-ozpe:~/FiltersTest/Test>
>> ../netcdf-c-4.5.0/ncdump/nccopy sample.nc sample_nccopy45.nc
>>
>> /Deconding/
>> /CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ... /
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 2/
>> /Decompression successfull/
>> /Decode End: 3600/
>>
>> /Deconding/
>> /CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, ... /
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 1/
>> /Decompression successfull/
>> /Decode End: 1800/
>>
>>
>> * //Nccopy in NetCDF-c 4.6: the decoding functionality in the filter is
>> invoked once for each variable when reading in_file. The encoding
>> functionality in the filter is invoked once for each variable when writing
>> out_file
>>
>> jrgu@linux-ozpe:~/FiltersTest/Test> nccopy sample.nc
>> sample_nccopy46.nc
>> /Deconding/
>> /CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ... /
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 2/
>> /Decompression successfull/
>> /Decode End: 3600/
>>
>> /Deconding/
>> /CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, .../
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 1/
>> /Decompression successfull/
>> /Decode End: 1800/
>>
>> /Encoding/
>> /CD VALUES 2, 1, 60, 30, 2, 1, 60, 30, 16, 1, .../
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 2/
>> /Pixel: 1800/
>> /ERROR: Error in jpeglsCompress: Parameter values are not
>> supported by CharLS./
>> /Compression failed/
>> /ERROR: Error during the JPEG-LS compression of the dataset./
>> /Encode End -> Result: 0/
>>
>> /Encoding/
>> /CD VALUES 1, 1, 60, 30, 1, 1, 60, 30, 8, 1 /
>> /Components: 1, Lines: 60, Samples: 30/
>> /DataBytes: 1/
>> /Pixel: 1800/
>> /ERROR: Error in jpeglsCompress: Parameter values are not
>> supported by CharLS./
>> /Compression failed/
>> /ERROR: Error during the JPEG-LS compression of the dataset./
>> /Encode End -> Result: 0/
>>
>> Cheers,
>> Joaquín
>>
>
