Re: [thredds] Time Partition Syntax

• To: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
• Subject: Re: [thredds] Time Partition Syntax
• From: Kevin Manross <manross@xxxxxxxx>
• Date: Wed, 17 Apr 2013 14:43:16 -0600

On your tip, Roy, I placed a bunch of files in a scratch directory.

/glade/nwsc_scratch/manross/ds083.2/grib1

I verified that I can write to that directory as another user.


I then tried the following catalog.xml config:

   <metadata inherited="true">
     <serviceName>all</serviceName>
     <dataType>Grid</dataType>
     <dataFormat>GRIB-1</dataFormat>
       <documentation type="summary">LOCAL FNL's TO TEST TIME PARTITION
       </documentation>
     </metadata>
   <collection spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/"
               timePartition="1 day"
               dateFormatMark="fnl_#yyyyMMdd_HH_mm#"
   />
   <update startup="true" rescan="0 0 0 * * ? *"/>
   <gribConfig datasetTypes="Best Files" />
 </featureCollection>

which resulted in the following error in featureCollectionScan.log

java.lang.NullPointerException

    at java.util.Arrays.mergeSort(Arrays.java:1283)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.mergeSort(Arrays.java:1294)
    at java.util.Arrays.sort(Arrays.java:1223)
    at java.util.Collections.sort(Collections.java:176)

    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)

    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:164)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)

    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)

   <metadata inherited="true">
     <serviceName>all</serviceName>
     <dataType>Grid</dataType>
     <dataFormat>GRIB-1</dataFormat>
       <documentation type="summary">LOCAL FNL's TO TEST TIME PARTITION
       </documentation>
     </metadata>

               timePartition="1 day"
   />
   <update startup="true" rescan="0 0 0 * * ? *"/>
   <gribConfig datasetTypes="Best Files" />
 </featureCollection>


Which resulted in a different error:

java.lang.NullPointerException

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)

    at java.lang.Thread.run(Thread.java:679)


So I'm at a loss of what to do.

Thanks!

-kevin.


On 4/17/13 9:31 AM, Roy Mendelssohn - NOAA Federal wrote:

That is not where they write.  It is literally where the data is stored.  Look 
in your data directory and see if there are .ncx and the other is .gXX (I can't 
remember the file ender).  This threw me also for awhile.  I had to change 
permissions on my data directories.  I don't know that this is your error, but 
it is worth checking.

-Roy
On Apr 17, 2013, at 8:27 AM, Kevin Manross <manross@xxxxxxxx> wrote:

Hi Roy,

Thanks for the reply and it's a good question.

I believe the directory
/usr/share/tomcat6/content/thredds/cache/grib/
does have permissions to write.  When I take away the time partition and merely 
scan the directory path, it has been writing files to that directory for a few 
weeks.  When I add the time partition, the specific file

/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx
is never written.

-kevin.

On 4/17/13 9:20 AM, Roy Mendelssohn - NOAA Federal wrote:

Hi Kevin:

Just out of curiosity does your data directory  (not your catalog directory) 
have permissions for tomcat to write files?  The new featureCollection writes 2 
files for each dataset.  If it can't write, the entire thing fails.

Don't know if that is causing your error, but you might check.

-Roy

On Apr 17, 2013, at 8:11 AM, Kevin Manross
<manross@xxxxxxxx>
  wrote:

Got a (hopefully) quick question regarding setting up the Time Partition for a 
GRIB collection.  I don't seem to be making headway with it and I hoped someone 
might be able to help me sort it out.

I have files in a path structure like:

/glade/p/rda/data/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_mm[_c]

Where the _c shows up after a certain date.

I have my collection set up as follows:

spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                timePartition="1 day"
                dateFormatMark="fnl_#yyyyMMdd_HH_mm#"

When I go to the collection folder, I'm getting a 404 on the catalog.  The 
featurecollectionScan.log file shows a null pointer exception:

[2013-04-12T09:14:37.958-0600] INFO  thredds.inventory.CollectionUpdater: 
Schedule startup scan for /glade/p/rda/data/ds083.2/grib1/**/fnl_* at Fri Apr 
12 09:14:47 MDT 2013
[2013-04-12T09:14:37.968-0600] INFO  thredds.inventory.CollectionUpdater: 
Schedule recurring scan for /glade/p/rda/data/ds083.2/grib1/**/fnl_* cronExpr=0 
0 0 * * ? *
[2013-04-12T09:14:57.443-0600] INFO  thredds.inventory.MFileCollectionManager: 
NCEP_FNL : was scanned MCollection{name='/glade/p/rda/data/ds083.2/grib1', 
dirName='/glade/p/rda/data/ds083.2/grib1', wantSubdirs=true, 
ff=WildcardMatchOnPath{wildcard=null regexp=fnl_*}}
[2013-04-12T09:15:01.816-0600] INFO  thredds.inventory.MFileCollectionManager: 
NCEP_FNL : was scanned MCollection{name='/glade/p/rda/data/ds083.2/grib1', 
dirName='/glade/p/rda/data/ds083.2/grib1', wantSubdirs=true, 
ff=WildcardMatchOnPath{wildcard=null regexp=fnl_*}}
[2013-04-12T09:15:01.818-0600] INFO  
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder: TimePartitionBuilder createIndex 
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx
[2013-04-12T09:15:24.270-0600] ERROR thredds.catalog.InvDatasetFcGrib: Error 
making catalog for FNLCollection
java.lang.NullPointerException
     at 
thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:454)
     at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:378)
     at 
thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1469)
     at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1432)
     at 
thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
     at 
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at 
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
     at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
     at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)


I've tried a number of different combinations:

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="#fnl_#yyyyMMdd_HH_mm"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_#yyyyMMdd_HH_mm#"
                timePartition="1 day"


    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/'fnl_'#yyyyMMdd_HH_mm#"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="'fnl_'#yyyyMMdd_HH_mm#"
                timePartition="1 day"

    <collection spec="/glade/p/rda/data/ds083.2/grib1/**/fnl_*"
                dateFormatMark="#fnl_#yyyyMMdd_HH_mm"
                timePartition="1 day"


And in (almost) each case, when I click on the catalog I an entry in the 
featureCollectionScan.log file reads:

[2013-04-12T14:17:33.591-0600] INFO  
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder: TimePartitionBuilder createIndex 
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx

and I get an Internal Server Error (500) from the website.

The file:  
/usr/share/tomcat6/content/thredds/cache/grib/glade-p-rda-data-ds083.2-grib1-NCEP_FNL.ncx
 is never written

If I then click reload on the page, I get the NullPointerException - I'm 
guessing because that file isn't written.

Any idea what I'm doing wrong here?

-kevin.


--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218

Email:manross@xxxxxxxx

Web:
http://rda.ucar.edu

_______________________________________________
thredds mailing list

thredds@xxxxxxxxxxxxxxxx

For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/

**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail:
Roy.Mendelssohn@xxxxxxxx
  (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www:
http://www.pfeg.noaa.gov/


"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx
Web:http://rda.ucar.edu

**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu

• Follow-Ups:
  • Re: [thredds] Time Partition Syntax
    • From: Kevin Manross

• References:
  • [thredds] Time Partition Syntax
    • From: Kevin Manross
  • Re: [thredds] Time Partition Syntax
    • From: Kevin Manross