Re: [thredds] NetCDF Subset service

Hi Terry,

I did some digging and found a related issue on the GDAL Trac site [1]. It
seems like the problem is the "elevation:_ChunkSizes" attribute: in the
original dataset it is "1, 946" and in the NCSS version it is "2631, 946".
The ticket indicates that GDAL prefers the former.

As it turns out, a recent change to THREDDS addresses this issue by
attempting to maintain the old chunk sizes when writing new NetCDF-4 data.
As a test, I fired up the most recent development version of THREDDS
(4.5.1) [2], and ran your original dataset through NCSS. I've attached the
result to this email (busselton-final-ncss4.5.1.nc4). Can you run your code
on it to see if the issue persists?

Cheers,
Christian Ward-Garrison

[1] https://trac.osgeo.org/gdal/ticket/5082
[2] http://www.unidata.ucar.edu/software/thredds/current/tds/TDS.html


On Thu, Jul 24, 2014 at 9:54 PM, <Terry.Rankine@xxxxxxxx> wrote:

> Hi Christian
>
>
>
> *I think *Thredds has destroyed my PROJCS data which doesn’t allow my
> code to rewarp it into a different projection.
>
>
>
> busselton-final-meta.nc-thredds-ncss.nc4 -> subset service “request all”
>
> busselton-final-meta.nc -> original file from HTTP service.
>
>
>
>
> http://vhirl-dev.csiro.au/thredds/catalog/dem/catalog.html?dataset=testDatasetScan/busselton-final-meta.nc
>
>
>
>
>
>
>
> ran110@vhirl-dev:~/testing$ gdalwarp
> busselton-final-meta.nc-thredds-ncss.nc4 -t_srs "+proj=latlong
> +datum=WGS84" -co "COMPRESS=DEFLATE" -r cubic output.wgs84.cubic.nc  -wo
> OPTIMIZE_SIZE=TRUE -of netCDF -dstnodata -9999
>
> Creating output file that is 1152P x 2597L.
>
> Warning 5: NOTICE: Format set to NC4C because compression is set to
> DEFLATE.
>
> Processing input file busselton-final-meta.nc-thredds-ncss.nc4.
>
> Using internal nodata values (e.g. -9999) for image
> busselton-final-meta.nc-thredds-ncss.nc4.
>
> ERROR 1: nBlockYSize = 2631, only 1 supported when reading bottom-up
> dataset
>
> ERROR 1: busselton-final-meta.nc-thredds-ncss.nc4, band 1: IReadBlock
> failed at X offset 0, Y offset 0
>
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
>
>
>
> ran110@vhirl-dev:~/testing$ gdalwarp busselton-final-meta.nc -t_srs
> "+proj=latlong +datum=WGS84" -co "COMPRESS=DEFLATE" -r cubic
> output.wgs84.cubic.nc  -wo OPTIMIZE_SIZE=TRUE -of netCDF -dstnodata -9999
>
> Creating output file that is 1152P x 2597L.
>
> Warning 5: NOTICE: Format set to NC4C because compression is set to
> DEFLATE.
>
> Processing input file busselton-final-meta.nc.
>
> Using internal nodata values (e.g. -9999) for image
> busselton-final-meta.nc.
>
> 0...10...20...30...40...50...60...70...80...90...100 - done.
>
> ran110@vhirl-dev:~/testing$
>
>
>
>
>
> HEADER INFO
>
>
>
> ran110@vhirl-dev:~/testing$ gdalinfo
> busselton-final-meta.nc-thredds-ncss.nc4
>
> Driver: netCDF/Network Common Data Format
>
> Files: busselton-final-meta.nc-thredds-ncss.nc4
>
> Size is 946, 2631
>
> Coordinate System is:
>
> PROJCS["unnamed",
>
>     GEOGCS["unknown",
>
>         DATUM["unknown",
>
>             SPHEROID["Spheroid",6378137,298.257223563]],
>
>         PRIMEM["Greenwich",0],
>
>         UNIT["degree",0.0174532925199433]],
>
>     PROJECTION["Transverse_Mercator"],
>
>     PARAMETER["latitude_of_origin",0],
>
>     PARAMETER["central_meridian",117],
>
>     PARAMETER["scale_factor",0.9996],
>
>     PARAMETER["false_easting",500],
>
>     PARAMETER["false_northing",10000],
>
>     UNIT["kilometre",1000,
>
>         AUTHORITY["EPSG","9036"]]]
>
> Origin = (189.659982684810018,6757.539391574399815)
>
> Pixel Size = (0.250000000000000,-0.250000000000000)
>
>
>
>
>
> VS
>
>
>
> ran110@vhirl-dev:~/testing$ gdalinfo  busselton-final-meta.nc
>
> Driver: netCDF/Network Common Data Format
>
> Files: busselton-final-meta.nc
>
> Size is 946, 2631
>
> Coordinate System is:
>
> PROJCS["UTM Zone 50, Southern Hemisphere",
>
>     GEOGCS["WGS 84",
>
>         DATUM["WGS_1984",
>
>             SPHEROID["WGS 84",6378137,298.257223563,
>
>                 AUTHORITY["EPSG","7030"]],
>
>             TOWGS84[0,0,0,0,0,0,0],
>
>             AUTHORITY["EPSG","6326"]],
>
>         PRIMEM["Greenwich",0,
>
>             AUTHORITY["EPSG","8901"]],
>
>         UNIT["degree",0.0174532925199433,
>
>             AUTHORITY["EPSG","9108"]],
>
>         AUTHORITY["EPSG","4326"]],
>
>     PROJECTION["Transverse_Mercator"],
>
>     PARAMETER["latitude_of_origin",0],
>
>     PARAMETER["central_meridian",117],
>
>     PARAMETER["scale_factor",0.9996],
>
>    PARAMETER["false_easting",500000],
>
>     PARAMETER["false_northing",10000000],
>
>     UNIT["METERS",1]]
>
> Origin = (189659.982684810005594,6757539.391574399545789)
>
> Pixel Size = (250.000000000000000,-250.000000000000000)
>
>
>
>
>
> *From:* Christian Ward-Garrison [mailto:cwardgar@xxxxxxxx]
> *Sent:* Friday, 25 July 2014 7:49 AM
> *To:* Rankine, Terry (Mineral Resources, Kensington)
> *Cc:* thredds@xxxxxxxxxxxxxxxx
> *Subject:* Re: [thredds] NetCDF Subset service
>
>
>
> Hi Terry,
>
> You might expect that when you request all of the data in a dataset, NCSS
> will simply return the original dataset (no subsetting required). That's
> not how NCSS works; instead, it will always create a new dataset, and copy
> over the requested data (which could potentially be everything) from the
> original.
>
> In this scenario, you might notice that the original and new datasets are
> *structurally* different (i.e. different attributes, scale factors, etc),
> but they are *semantically* equal (i.e. they contain the same data, though
> it might be represented differently).
>
> As far as I can tell, your 2 example files are semantically equal. Are you
> struggling with structural equality issues?
>
> Cheers,
>
> Christian
>
>
>
> On Mon, Jul 21, 2014 at 12:46 AM, <Terry.Rankine@xxxxxxxx> wrote:
>
> Hi Guys
>
>
>
> Is there any other users currently using the netCDF subset service? I am
> having quite a few issues with it and would like to check ‘it’s not just me’
>
>
>
> This:
>
> http://vhirl-dev.csiro.au/thredds/fileServer/dem/busselton-final.nc
>
>
>
> is very very different to this:
>
>
> http://vhirl-dev.csiro.au/thredds/ncss/grid/dem/busselton-final.nc?var=elevation&north=-29.2729&west=113.5915&east=116.2396&south=-35.2422&horizStride=1&accept=netcdf4
>
>
>
> How are others handling THREDDS and CRS changes?
>
>
>
> Regards
>
> Terry
>
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
>

Attachment: busselton-final-ncss4.5.1.nc4
Description: Binary data

  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: