Re: [thredds] Time Partition Syntax


Hi Roy,

Thanks for the reply and it's a good question.

I believe the directory

/usr/share/tomcat6/content/thredds/cache/grib/

does have permissions to write. When I take away the time partition and merely scan the directory path, it has been writing files to that directory for a few weeks. When I add the time partition, the specific file

/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx

is never written.

-kevin.

On 4/17/13 9:20 AM, Roy Mendelssohn - NOAA Federal wrote:
Hi Kevin:

Just out of curiosity does your data directory  (not your catalog directory) 
have permissions for tomcat to write files?  The new featureCollection writes 2 
files for each dataset.  If it can't write, the entire thing fails.

Don't know if that is causing your error, but you might check.

-Roy

On Apr 17, 2013, at 8:11 AM, Kevin Manross <manross@xxxxxxxx> wrote:

Got a (hopefully) quick question regarding setting up the Time Partition for a 
GRIB collection.  I don't seem to be making headway with it and I hoped someone 
might be able to help me sort it out.

I have files in a path structure like:

/glade/p/rda/data/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_mm[_c]

Where the _c shows up after a certain date.

I have my collection set up as follows:

spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                timePartition="1 day"
                dateFormatMark="fnl_#yyyyMMdd_HH_mm#"

When I go to the collection folder, I'm getting a 404 on the catalog.  The 
featurecollectionScan.log file shows a null pointer exception:

[2013-04-12T09:14:37.958-0600] INFO  thredds.inventory.CollectionUpdater: 
Schedule startup scan for /glade/p/rda/data/ds083.2/grib1/**/fnl_* at Fri Apr 
12 09:14:47 MDT 2013
[2013-04-12T09:14:37.968-0600] INFO  thredds.inventory.CollectionUpdater: 
Schedule recurring scan for /glade/p/rda/data/ds083.2/grib1/**/fnl_* cronExpr=0 
0 0 * * ? *
[2013-04-12T09:14:57.443-0600] INFO  thredds.inventory.MFileCollectionManager: 
NCEP_FNL : was scanned MCollection{name='/glade/p/rda/data/ds083.2/grib1', 
dirName='/glade/p/rda/data/ds083.2/grib1', wantSubdirs=true, 
ff=WildcardMatchOnPath{wildcard=null regexp=fnl_*}}
[2013-04-12T09:15:01.816-0600] INFO  thredds.inventory.MFileCollectionManager: 
NCEP_FNL : was scanned MCollection{name='/glade/p/rda/data/ds083.2/grib1', 
dirName='/glade/p/rda/data/ds083.2/grib1', wantSubdirs=true, 
ff=WildcardMatchOnPath{wildcard=null regexp=fnl_*}}
[2013-04-12T09:15:01.818-0600] INFO  
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder: TimePartitionBuilder createIndex 
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx
[2013-04-12T09:15:24.270-0600] ERROR thredds.catalog.InvDatasetFcGrib: Error 
making catalog for FNLCollection
java.lang.NullPointerException
     at 
thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:454)
     at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:378)
     at 
thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1469)
     at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1432)
     at 
thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
     at 
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at 
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
     at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
     at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)


I've tried a number of different combinations:

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="#fnl_#yyyyMMdd_HH_mm"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_#yyyyMMdd_HH_mm#"
                timePartition="1 day"


    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/'fnl_'#yyyyMMdd_HH_mm#"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="'fnl_'#yyyyMMdd_HH_mm#"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="#fnl_#yyyyMMdd_HH_mm"
                timePartition="1 day"


And in (almost) each case, when I click on the catalog I an entry in the 
featureCollectionScan.log file reads:

[2013-04-12T14:17:33.591-0600] INFO  
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder: TimePartitionBuilder createIndex 
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx

and I get an Internal Server Error (500) from the website.

The file:  
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx
 is never written

If I then click reload on the page, I get the NullPointerException - I'm 
guessing because that file isn't written.

Any idea what I'm doing wrong here?

-kevin.


--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx
Web:http://rda.ucar.edu
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.


--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: