Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work

  • To: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Subject: Re: [thredds] TDS 4.6 - I still can't get GRIB aggregations to work
  • From: John Caron <caron@xxxxxxxx>
  • Date: Thu, 9 Apr 2015 12:57:31 -0600
Sorry, lets try that again:

Hi Roy:

java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on
FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3
ParseException:Unparseable date: "ssure-2013"

So it looks like the index files (ncx3) are in the data directory, and
those need to get excluded. If all of your files end in "pres", you could
use:

  <collection spec="/u00/FNMOC/NAVGEM/pressure/**/.*pres$"
timePartition="directory"
 olderThan="5 min"/>

If not all of your files end in "pres", then we need to get more clever
with the regexp.

Note that GRIB collections no longer need the date filename parser (unless
you are using timePartition=<time duration>) so we dont need the #
rigamorole:

   US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$

So its a litttle easier to write the regexp that will get just the files
you want.

Another thing one could do is to place the index files in a seperate
directory. But I assume you have chosen to keep the indexes with the data?

John





On Thu, Apr 9, 2015 at 12:34 PM, Roy Mendelssohn - NOAA Federal <
roy.mendelssohn@xxxxxxxx> wrote:

> Hi All:
>
> I have a bunch of FNMOC Grib1 pressure field.  My aggregations work fine
> in TDS 4.3.x, but I have not been able to get them to work in either 4.5 or
> now 4.6.  I am probably being brain-dead or missing something obvious, but
> here is the error message I am getting:
>
>
> > 2015-04-09T08:29:13.343 -0700 INFO  - Schedule startup scan force=always
> for 'FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' at
> 2015-04-09T08:29:16.339-0700
> > 2015-04-09T08:29:16.817 -0700 ERROR - UpdateCollectionJob.execute
> startup failed collection=FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure
> > java.lang.RuntimeException: SimpleDateFormat = yyyyMMddHH fails on
> FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure-201302.ncx3
> ParseException:Unparseable date: "ssure-2013"
> >       at
> ucar.nc2.units.DateFromString.getDateUsingCompleteDateFormatWithOffset(DateFromString.java:253)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.units.DateFromString.getDateUsingDemarkatedCount(DateFromString.java:99)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> thredds.inventory.DateExtractorFromName.getCalendarDateFromPath(DateExtractorFromName.java:86)
> ~[cdm-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.PartitionCollectionMutable$Partition.<init>(PartitionCollectionMutable.java:364)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.PartitionCollectionMutable.addPartition(PartitionCollectionMutable.java:411)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribPartitionBuilder.createPartitionedIndex(GribPartitionBuilder.java:139)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updatePartition(GribCdmIndex.java:376)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updateDirectoryCollectionRecurse(GribCdmIndex.java:465)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.updateGribCollection(GribCdmIndex.java:341)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> ucar.nc2.grib.collection.GribCdmIndex.openGribCollection(GribCdmIndex.java:673)
> ~[grib-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:151)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.checkState(InvDatasetFeatureCollection.java:309)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.update(InvDatasetFeatureCollection.java:330)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.catalog.InvDatasetFeatureCollection.sendEvent(InvDatasetFeatureCollection.java:226)
> ~[tdcommon-4.6.0.jar:4.6.0]
> >       at
> thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:242)
> [cdm-4.6.0.jar:4.6.0]
> >       at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
> [quartz-2.2.0.jar:?]
> >       at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:560)
> [quartz-2.2.0.jar:?]
> > 2015-04-09T08:34:00.754 -0700 INFO  - FeatureCollection added =
> FeatureCollectionConfig name ='FNMOC NAVGEM 0.5-degree 6-hourly Pressure'
> collectionName='FNMOC_NAVGEM_0.5-degree_6-hourly_Pressure' type='GRIB1'
> >
>  
> spec='/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$'
> >   olderThan =5 min
> >   timePartition =directory
> >   updateConfig =UpdateConfig{userDefined=true, recheckAfter='null',
> rescan='null', triggerOk=true, updateType=always}
> >   tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null',
> rescan='null', triggerOk=true, updateType=always}
> >   ProtoConfig{choice=Penultimate, change='null', param='null',
> outerNcml='null', cacheAll=true}
> >   hasInnerNcml =false
> >   GribConfig{datasets=[Best, Latest], useGenType=false,
> useTableVersion=false, intvMerge=true, useCenter=false,
> filesSortIncreasing=true}
>
>
> Here is the relevant part of  the catalog:
>
> >      <featureCollection name="FNMOC NAVGEM 0.5-degree 6-hourly Pressure"
> featureType="GRIB1" harvest="true" path="FNMOC/navgem/pressure/6hr">
> >        <metadata inherited="true">
> >          <dataType>GRID</dataType>
> >           <dataFormat>GRIB-1</dataFormat>
> >           <serviceName>all</serviceName>
> >           <documentation type="summary">
> >               FNMOC 1-degree pressure fields from the NOGAPS model
> >            </documentation>
> >        </metadata>
> >
> >        <collection
> spec="/u00/FNMOC/NAVGEM/pressure/**/US058GMET-GR1mdl.0018_0056_00000F0..#yyyyMMddHH#_0102_000000-000000pres$"
> >            timePartition="directory"    olderThan="5 min"/>
> >
>
>
>
> This is my test directory structure:
>
> > root@localhost pressure]# pwd
> > /u00/FNMOC/NAVGEM/pressure
> > [root@localhost pressure]# ls
> > 201302
> >
>
>
> Here are typical files  (note some have F0FL and some F0OF which why the
> "F0..” in the time string
>
> > US058GMET-GR1mdl.0018_0056_00000F0RL2013021400_0102_000000-000000pres
> > US058GMET-GR1mdl.0018_0056_00000F0OF2013021306_0102_000000-000000pres
>
>
> Some things on the internet suggest that files have to have “.grib1” at
> the end,  Is this true?  If not, if anyone can see what my error is I sure
> would appreciate it.  This is why I haven’t upgraded.
>
> Thanks,
>
> -Roy
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: