Thanks, Jon. Apparently I missed Don's message regarding using the
GridDataset class
, was it sent to this mailing list?
Nevertheless
I was made aware of the usefulness class yesterday by a colleague (Jedi
Steve Ansari) and it's become clear that I've been doing all manner of
things in my code the hard way. It appears that by using GridDataset and
associated interfaces/classes I can get a lot of useful magic right out of
the box. I'm currently refactoring my code to leverage this approach.
Your other points below are well taken as well. Thanks for such a helpful
response.
--James
On Thu, Aug 14, 2014 at 4:04 AM, Jon Blower <j.d.blower@xxxxxxxxxxxxx>
wrote:
> Hi James,
>
> Short answer - GridDataset.getCalendarDateRange() may do the job as Don
> suggests. (Don, thanks for this! I’d forgotten that NetCDF-Java had started
> to support Joda. It could be that we can remove this code from ncWMS if all
> the features are now supported in ncj. Maybe John Caron can advise on this.)
>
> But in answer to your specific questions:
>
> 1. There is a CF compliance checker online (can’t remember the URL but
> you can Google it). However, this can’t check everything, so you should
> regard it as “necessary but not sufficient”. I think Unidata has a service
> where you can check that your data file is readable by the Common Data
> Model - this is more useful in practice for testing compliance with
> Java-NetCDF, but I can’t remember where it lives.
>
> 2. I’m not familiar with the DateUnit class - maybe I misunderstood
> this. Does this correctly parse UDUNITS strings like “days since
> 1970-01-01”? Sorry for the noise if so.
>
> 3. Having a vertical axis is not part of CF-compliance - but if your
> data do have a vertical axis it should be tagged up correctly with CF
> metadata. This is easy to get wrong, and a large proportion of metadata
> problems (in my experience) are due to incorrectly-formatted vertical axes.
>
> Cheers,
> Jon
>
> --
> Dr Jon Blower,
> Technical Director, Reading e-Science Centre,
> MELODIES project coordinator,
> School of Mathematical and Physical Sciences,
> University of Reading
> Tel: +44 118 378 5213
> Email: j.d.blower@xxxxxxxxxxxxx
>
> On 13 Aug 2014, at 20:36, James Adams - NOAA Affiliate <
> james.adams@xxxxxxxx> wrote:
>
> Thanks, Jon. Some follow up comments are inline below.
>
> On Wed, Aug 13, 2014 at 12:26 PM, Jon Blower <j.d.blower@xxxxxxxxxxxxx>
> wrote:
>
>> Hi James,
>>
>> I understand not wanting too many 3rd-party dependencies, but time
>> handling is tricky and you will just end up finding the same issues as
>> everyone else has found. Unless your data have very specific metadata that
>> meets your code’s expectations, you will sooner or later find that a
>> perfectly CF-compliant file doesn’t work with your code. This is the
>> trade-off I guess! ;-)
>>
>>
> You make a very good point, something I need to bear in mind if I ever
> want to generalize my code to handle more types of data sets. However this
> is code I'm writing to use within various other programs I've written to
> handle a specific type of NetCDF data set, and I just want to make sure I'm
> following best practices as far as is practical. Hopefully I'll be able to
> harden the code further to accommodate other types of NetCDF data sets in
> the future, but for now I'm only dealing with a specific type of file, one
> that I've written which is more or less CF-compliant (from what I can tell)
> -- is there a program I can use to check that, and/or there various
> CF-compliance levels? This is probably a topic for another thread, as it's
> not Java related.
>
>
>> A few comments on your code:
>>
>> 1. It relies on the time variable being called “time”. The correct way
>> (for the CF conventions) to identify a time variable is to look at its
>> units. (Alternatively the NetCDF-Java libraries can do this for you, I
>> think).
>>
>>
> You're correct, I'm assuming the presence of a variable named "time".
> However I'd rather do things the correct way -- can anyone point me to an
> example of how this is typically done using NetCDF-Java?
>
>
>> 2. It doesn’t appear to parse the units string correctly, assuming that
>> the units string is UDUNITS-compliant. Maybe it’s pseudocode though, in
>> which case fine.
>>
>
> This is good to know, as I've used this approach in other programs with
> success, but maybe I've just been lucky? Here's how it looks:
>
> Attribute inputTimeUnits =
> inputTimeVariable.findAttributeIgnoreCase("units");
> DateUnit dateUnit = new
> DateUnit(inputTimeUnits.getStringValue());
> Date initialDate = dateUnit.makeDate(inputTimeArray.getInt(0));
> Calendar calendar = Calendar.getInstance();
> calendar.setTimeZone(TimeZone.getTimeZone("GMT"));
> calendar.setTime(initialDate);
> int startYear = calendar.get(Calendar.YEAR);
>
> calendar.setTime(dateUnit.makeDate(inputTimeArray.getInt(numberOfInputMonths
> - 1)));
> int endYear = calendar.get(Calendar.YEAR);
>
> Are you saying that it's better to use something other than the DateUnit
> class for the units parsing? If so then what's the best way forward?
>
>
>> 3. It would only work for Gregorian dates, no other calendar systems.
>>
>>
> Agreed, but this is sufficient for my limited purposes.
>
> 4. No time zone support, as you point out
>>
>> (By the way, if it makes you feel any better, the ncWMS code is mature
>> and has been used for years without change, so it would seem to work OK. If
>> you didn’t want the whole JAR, you could just factor out the classes you
>> need, or cannibalise the code. It’s all open source.)
>>
>>
> I'd like to use something like ncWMS since it appears that you guys
> have done a lot of heavy lifting already, but the data set files my code is
> processing may not be sufficient (as you've pointed out earlier) since
> there's no vertical axis. Maybe this is a place for improvement in my
> process, i.e. I should add a vertical axis in order to make the NetCDF
> files easier to use by other tools. Is having a vertical axis part of
> CF-compliance? If so I'd like to make some updates to my NetCDF generation
> code, but this will have downstream effects that probably won't fly for
> now. :(
>
>
>> Cheers,
>> Jon
>>
>> --
>> Dr Jon Blower,
>> Technical Director, Reading e-Science Centre,
>> MELODIES project coordinator,
>> School of Mathematical and Physical Sciences,
>> University of Reading
>> Tel: +44 118 378 5213
>> Email: j.d.blower@xxxxxxxxxxxxx
>>
>> On 13 Aug 2014, at 15:41, James Adams - NOAA Affiliate <
>> james.adams@xxxxxxxx> wrote:
>>
>> Thanks for your help, Jon. I'm reticent to cook third-party JAR
>> dependencies into my code for a number of reasons if I can avoid it. It's
>> looking like there's no real best practice for this sort of thing and
>> everyone just rolls their own, so maybe I should just follow suit? The
>> NetCDF-Java API is all over the place, at least based on what I can see in
>> the Javadoc. (For example there's a CalendarDateUnit class with no
>> constructor and flotsam comments from you and Bob Simon included within the
>> main description section -- is this class useful at all or recommended?)
>>
>> For now the below is what I've cooked up -- does this look
>> reasonable? It seems less than bulletproof (for example how do you best
>> account for time zone, etc.?) but maybe it's good enough for now, this code
>> isn't going to be part of a nuclear reactor control system. ;)
>>
>> Variable timeCoordinateVariable =
>> netcdfDataset.findVariable("time");
>> String timeUnits =
>> timeCoordinateVariable.findAttribute("units").toString();
>> ArrayInt timeArray = (ArrayInt) timeCoordinateVariable.read();
>> DateUnit dateUnit = new DateUnit(timeUnits);
>> Date initialDate = dateUnit.makeDate(timeArray.getInt(0));
>> DateTime initialDateTime = new
>> DateTime(initialDate.getTime(), GregorianChronology.getInstanceUTC());
>> int initialYear = dateTime.getYear();
>>
>> --James
>>
>> On Wed, Aug 13, 2014 at 3:02 AM, Jon Blower <j.d.blower@xxxxxxxxxxxxx>
>> wrote:
>>
>>> Hi James,
>>>
>>> The ncWMS libraries contain some high-level calls that might help here.
>>> The uk.ac.rdg.resc.edal.cdm.CdmUtils class is the one you want:
>>>
>>> NetcdfDataset nc = NetcdfDataset.open(“/path/to/my/netcdf/file”);
>>> GridDatatype grid = CdmUtils.getGridDatatype(nc, “myvar”);
>>> CoverageMetadata cm = CdmUtils.readCoverageMetadata(grid);
>>> List<DateTime> dateTimes = cm.getTimeValues();
>>>
>>> This gives a list of all the DateTime objects (these are from the
>>> joda-time library) in the time axis of the variable “myvar”. They will be
>>> referenced to a given Chronology (i.e. calendar system), which you can find
>>> through cm.getChronology().
>>>
>>> The main snag is that your variable has to be fully georeferenced,
>>> otherwise readCoverageMetadata() won’t work. A common problem here is that
>>> the vertical axis isn’t always well defined in NetCDF files.
>>>
>>> If you’re interested in this solution you can build a jar containing the
>>> above code:
>>>
>>> svn checkout svn://svn.code.sf.net/p/ncwms/code/trunk ncwms
>>> cd ncwms
>>> ant jar-for-THREDDS
>>>
>>> Hope this helps,
>>> Jon
>>>
>>>
>>> --
>>> Dr Jon Blower,
>>> Technical Director, Reading e-Science Centre,
>>> MELODIES project coordinator,
>>> School of Mathematical and Physical Sciences,
>>> University of Reading
>>> Tel: +44 118 378 5213
>>> Email: j.d.blower@xxxxxxxxxxxxx
>>>
>>>
>>>
>>
>>
>
>
