NOTE: The cf-satellite mailing list is no longer active. The list archives are made available for historical reasons.
Morning Martin, Sorry to hear that you were at the user conference but were not able to attend the presentation I gave. I've attached the presentation for information, and I welcome feedback on the subject matter from subscribers of this mailing list. The southMostLine, eastMostPixel, northMostLine, westMostPixel global attributes are defined to inform users who are used to using these parameters of the subarea ordered. This is particularly useful when users order a lot of data as it reminds them of the contents of the data. Yes I agree that this is not essential but adding meta-data like this even though it may only be useful to some users adds value to the data. Knowing the needs of your core users will help determine what type of meta-data to specify and your input is very useful. I agree that implementing the HRV in 2 arrays is a good way to organise the data. This avoids a lot of cells being 'empty' which would be the case if the data was stored in one array. We will be revising the MSG15 NetCDF data set to include more data stored in the native format so this will give us an opportunity to implement the HRV channel as well. Regarding your lat lon comment, Unidata provides Standard Coordinate Transforms to Geo-locating data onto a earth projection, see below: http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html This removes the need to store the lat and lon arrays in your data if you are using Unidata tools to visualise the data. For users who don't use these tools, it would require them to map these transforms to lat and lon i.e. they need to do extra programming to visualise the data. As I've mentioned before, usability of the data for all users is very important in our development. So even though the lat and lon uses space in the NetCDF file, it is provided for usability reasons. If you take a look at the presentation, we proposed the concept of tailoring NetCDF data sets to the user's needs. Whether the lat and lon arrays, coordination transform or any other method of geo-locating the data is included in the NetCDF file or not can be addressed here. We just need to get enough users requesting the implementation of tailored NetCDF files (hint :o). Regarding the band discussion and taking Tom's reinforcing Email into account (Thanks Tom ... might see you next year in Boulder if you are there to discuss more CF conventions with you) I think I understand what bands are now. Creating a 3D array indexed by a band or channel with lat and lon is not a problem. For me, I think the channel/band should be an coordinate variable so that it can be treated like 'level', "altitude" or "pressure level" by the Unidata tools for visualisation. I'm also in favour of the standard_name="channel" rather than "band" as this is clearer to me, of course this is only my input to this discussion. I'm don't know who you have been talking to at EUMETSAT regarding bands but now I know this is a user need, I can address this in our Format Advisory Group. Thanks for drawing my attention to this. In case anyone reading this thread have problems in understanding some of the term we are using. I've found the following tutorial to help understanding the basic NetCDF concepts and I hope this is useful. http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-tutorial.pdf Regards, Pete. -----Original Message----- From: Martin Raspaud [mailto:martin.raspaud@xxxxxxx] Sent: Monday, October 24, 2011 5:08 PM To: Peter Miu Cc: cf-satellite@xxxxxxxxxxxxxxxx Subject: Re: [cf-satellite] cf-satellite Digest, Vol 14, Issue 1 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24/10/11 15:35, Peter Miu wrote: > Hi Martin, Hi Pete, > Thanks for the fast feedback. Before I answer your questions, I would > just like to let you know that the EUMETSAT Data Centre is currently > planning the implementation NetCDF formats as the common delivery format for > all orderable data sets from the Archive. > > A pilot has already been performed for the level 1B and 2 products > from the Metop-A ASCAT instrument and these are available for user feedback > under the following URL: > > http://gsics.eumetsat.int/thredds/ascat.html > > The ASCAT NetCDF products were also presented to users in this year's > EUMETSAT user conference held in Oslo. > > The development of usable NetCDF formats is our top priority as we > want to ensure that all users (novice and expert) can immediately use > our data so it is very important that guidelines regarding best > practises, filename conventions, CF-Conventions and standards are > followed. A Format Advisory Group has been set up at EUMETSAT to > discuss these guidelines with a view to publishing them so that users > are aware how our data is organised and also to propose updates to the > CF-conventions for satellite data as work is needed here (hence the > existence of this mailing list! :o) I was at the EUMETSAT conference this year :) > Here are the answers to your questions: > > - Why are southMostLine, eastMostPixel, northMostLine, westMostPixel, and > numberOfChannels dimensions and not attributes ? > > Apart from the HRV channel, these values remain constant for all > channels in the data sets so we can be specify them as global > attributes. If the channel arrays store different 'sub-area' then we > would need these value to be stored as variable attributes for each channel > array. This is not the case for our NetCDF data set i.e. all channel arrays > hold the same area for the same time. > > Note, the NetCDF file follows the Unidata recommendation for > identifying the coordinates of the data in the channel arrays > (Coordinate System Object Model - Current Encodings). Longitude and latitude > arrays exist in the data set and their contents are used to geo-locating the > pixel counts for each channel onto a projection. > > See: http://www.unidata.ucar.edu/software/netcdf-java/CDM/ Ok. I don't really understand the interest of having these dimensions/attributes though, since I guess you will have to define lon/lats anyway for each different resolution... > - - How would you deal with the inclusion of the HRV channel ? > > We have not implemented the HRV channel in our NetCDF data sets as GSICS does > not use this channel for calibration. > > To implement this channel in our NetCDF data sets, we would just need > to add another longitude and latitude array specifying the location of each > HRV pixel counts and a 2D array holding the HRV pixel counts. Sounds good, but the HRV has 2 sub-areas: would you have a different variable for each or would you group the in one big channel ? (We implemented the latter, since the netcdf compression works well in this case) > - - Are the longitudes and latitudes values different from what a "vertical > perspective" with right parameters would provide ? If they do, why ? If they > don't, why not include a grid-mapping definition ? > > Not sure what you mean here, perhaps the following explain it: > > For the pixel count in ch1( y, x ), it is located at latitude lat( y, x ) and > longitude lon( y, x ). > By assigning the lat and lon as the coordinates system for any channel array, > applications know how to geo-locate its contents. > > float lat(y,x); > float lon(y,x); > int ch1(y,x); > ch1:coordinates="lat lon"; > > int ch2(y,x); > ch2:coordinates="lat lon"; My question was if the lon/lats differ from what one would get from running proj.4 with parameters like +proj=geos +lon0=0 +a=... etc. If one would get the same lon/lat data, then I would recommend to record also these parameters in a grid mapping variable. In this case, the lon/lats arrays would even become optional. > - - On the cf-satellite list, we have long been discussing "band"s. Did you > consider using those ? > > Sorry, I don't know what bands are. Well, this is a bit sad :(. We have been working quite a lot on this list toward having band variables that would correctly describe satellite channels (or bands). My colleagues and I have tried to drag Eumetsat's attention to this list and the work done here on several occasions. Tom Whittaker was also at the Eumetsat conference last year in Cordoba to present CF conventions and the possibility to apply these to satellite. But apparently that was not enough... You can read more on the band variable and related questions here: http://www.unidata.ucar.edu/mailing_lists/archives/cf-satellite/2011/thread.html > - - Did you consider grouping channels in 3d arrays instead of having them > separated ? > > I'm not sure what value a 3D arrays would give to the data. > Applications like the Unidata can plot the arrays on top of each other if > needed. > > As mentioned above, we want our data to be immediately usable by all > users and by following guidelines, we add values to the data as the > contents can be examined and plotted by existing free NetCDF applications > without any further processing by the users. > > - - Could you provide in the file calibration coefficients needed to convert > radiances to reflectances and brightness temperatures ? > > Yes we can add any variable attribute to the channel arrays to support the > needs of our users. > The power of NetCDF is as long as we don't remove or reorganise any > attributes and/or arrays, existing users and applications will not > 'break' by the change. The only concern here is we need to ensure > that a standard convention is followed for representing measurements > that can be stored in different units (e.g. temperature can be stored as > Celsius, Fahrenheit, Kelvin, ... ). What unit is used should be based on the > needs of the 'core' users and this unit with a standard name should be > included in the CF-conventions. That would be nice :) Thanks a lot. Best regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOpX9pAAoJEBdvyODiyJI4mC0IAK/6m0+TWMtaVcRChVGLJ9VX OY4OsSZQ+dceW4+YWGymBRUInUWChURdvy4eLCX9lPiz2TgOXbJkyvVcuK2iULtp 6oYTX+Fn3g3LhDM118rlYe5CApgFcg0bA62hqwXoM0ksrdQ6cbt1tz6aSV1kpkbL 4tCJrmLLen8mcYF3fSsH1xpKli5NUXEOwcf4jDWg9ieQAwic4RuD8bky1rshK3NS DDJeVjkKETE9VBMsseupWmptuWWG7zzFH9qsMatHjn8tYixTfezwaURu3C8n00Fm ugwHLO1uUkCb+jK7Ga1Kikftg7hFlATMTgLd1WRpZHyO0LqbbrgiLmvRHbsVYV0= =waU8 -----END PGP SIGNATURE-----
Attachment:
Data_Centre_NetCDF_Pilot_Implementation_Presentation.pdf
Description: Data_Centre_NetCDF_Pilot_Implementation_Presentation.pdf
cf-satellite archives: