Re: [thredds] Time Partition Syntax
Here's an update on attempting a Time Partition for model analyses that are in GRIB1:
I have created a local data structure for testing where I can write to the file directories (until I can get permission to write to the file directories of our main archive). The structure is as follows: 

/glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c
When I found the message thread : http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html
I took the advice mentioned therein and grabbed the 4.3.17 version of the thredds war, and set my catalog.xml config as follows: 

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit your server!" 
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
        xmlns:xlink="http://www.w3.org/1999/xlink";>

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" /> 
    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
    <service name="wms" serviceType="WMS" base="/thredds/wms/" />
<service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /> 
  </service>

  <!--datasetRoot path="FNL" location="content/ds083.2" /-->

  <featureCollection name="LOCAL_FNL"
                     path="LOCALFNL"
                     harvest="true"
                     featureType="GRIB">

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <documentation type="summary">LOCAL FNLS FOR TESING
      </documentation>
      <dataFormat>GRIB-1</dataFormat>
    </metadata>

    <collection
spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$"
        timePartition="1 day"
        dateFormatMark="#fnl_#yyyyMMdd_HH"
        />

    <update startup="true" rescan="0 0 0 * * ? *"/>
    <gribConfig datasetTypes="Best Files"/>

  </featureCollection>
</catalog>
Upon restarting tomcat, the directory was scanned and I was met with numerous "Too many open files" errors:
2013-04-29T13:12:17.795 -0600 [ 36341][ ] ERROR - fc.LOCAL_FNL - Grib1CollectionBuilder LOCAL_FNL-20130416 : reading/Creating gbx9 index for file /glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c failed java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9 (Too many open files) 
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
    at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199)
at ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.796 -0600 [ 36342][ ] ERROR - fc.LOCAL_FNL - Failed to open partition LOCAL_FNL-20130416 java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx (Too many open files) 
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.803 -0600 [ 36349][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.io.FileNotFoundException: /glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx (Too many open files) 
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
So I pared back the dataset from 3.5 years worth of 4 daily files to just the first month (still 4 files daily) of 2010-2013. This eliminated the "Too many open files" errors, and created the respective *.gbx9 and .ncx files, but threw a cast error:
2013-04-29T13:40:04.188 -0600 [ 108434][ ] INFO - fc.LOCAL_FNL - fnl_20130131_00_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx 2013-04-29T13:40:04.351 -0600 [ 108597][ ] INFO - fc.LOCAL_FNL - fnl_20130131_06_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx 2013-04-29T13:40:04.508 -0600 [ 108754][ ] INFO - fc.LOCAL_FNL - fnl_20130131_12_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx 2013-04-29T13:40:04.649 -0600 [ 108895][ ] INFO - fc.LOCAL_FNL - fnl_20130131_18_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx 2013-04-29T13:40:06.936 -0600 [ 111182][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.lang.ClassCastException: ucar.nc2.grib.GribCollection$VariableIndex cannot be cast to ucar.nc2.grib.TimePartition$VariableIndexPartitioned at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
And when trying to access my TDS from a browser, the catalog could not be made:
2013-04-29T13:40:33.330 -0600 [ 137576][ 7] INFO - threddsServlet - Remote host: 128.117.8.232 - Request: "GET /thredds/catalog/LOCALFNL/catalog.html HTTP/1.1" 2013-04-29T13:40:33.335 -0600 [ 137581][ 7] ERROR - fc.LOCAL_FNL - Error making catalog for LOCALFNL 
java.lang.NullPointerException
at thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453) at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377) at thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467) at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430) at thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) 
    ...


For grins, I tried changing the partition to

        timePartition="directory"
        dateFormatMark="#grib1/#yyyy"

And got the same ClassCastException.
If I'm missing something obvious, please let me know. I understand that the GRIB feature collection is still in flux, so please consider this some testing info. I'm able to make a GRIB collection without the time partition, but for file browsing, that partition would be a feature we would really want to incorporate. 

Cheers!

-kevin.

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: