Re: [thredds] Time Partition Syntax

  • To: Dan Swank - NOAA Affiliate <dan.swank@xxxxxxxx>
  • Subject: Re: [thredds] Time Partition Syntax
  • From: Kevin Manross <manross@xxxxxxxx>
  • Date: Thu, 02 May 2013 08:06:10 -0600

Hi Dan,

Thank you (and Tim Pugh) for the reminder to look at the JVM settings on the OS. Through John and Lansings's work, I have been able to get some of the time partitioning to work with my data set.

I am still getting the "too many files" error when trying to set up a daily partition, but no such errors with a monthly partition. I have yet to modify the OS JVM, and I'll try that today.

Much appreciated!

-kevin.


On 5/1/13 7:49 AM, Dan Swank - NOAA Affiliate wrote:
Just a quick plug...

Been closely working with Jahn Caron (up until he left for EGU), and
trying to get featureCollections working for all the major NCDC-NOMADS
 Archive datasets.  Both Grib1 & 2 ~ With moderate success.
Alot (but not all) of the issues I found were due to problems on our
own file system. Not permissions, but rather
zero-byte and/or corrupt/damaged GRIB records wrecking havoc
with the GRIB scanner and cause it to fail to create the master
time partition / index files.

  Caron added some modifications in the last few .wars to make
the GRIB scanner more tolerant, but as you mentioned there are still
lingering issues with it.

The "too many open files" sounds like you may need to adjust
an OS or JAVA env setting.   Not an expert on this and perhaps
someone else can chime in.  In short, it may not necessarily be
related to the featureCollectionScan.

     -Dan
   NCDC-NOMADS project




On Mon, Apr 29, 2013 at 4:17 PM, Kevin Manross <manross@xxxxxxxx <mailto:manross@xxxxxxxx>> wrote:


    Here's an update on attempting a Time Partition for model analyses
    that are in GRIB1:

    I have created a local data structure for testing where I can
    write to the file directories (until I can get permission to write
    to the file directories of our main archive).  The structure is as
    follows:

    /glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c


    When I found the message thread :
    http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html

    I took the advice mentioned therein and grabbed the 4.3.17 version
    of the thredds war, and set my catalog.xml config as follows:

    <?xml version="1.0" encoding="UTF-8"?>
    <catalog name="THREDDS Server Default Catalog : You must change
    this to fit your server!"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
    <http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0>
            xmlns:xlink="http://www.w3.org/1999/xlink";
    <http://www.w3.org/1999/xlink>>

      <service name="all" base="" serviceType="compound">
        <service name="odap" serviceType="OpenDAP"
    base="/thredds/dodsC/" />
        <service name="http" serviceType="HTTPServer"
    base="/thredds/fileServer/" />
        <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
        <service name="wms" serviceType="WMS" base="/thredds/wms/" />
        <service name="ncss" serviceType="NetcdfSubset"
    base="/thredds/ncss/grid/" />
      </service>

      <!--datasetRoot path="FNL" location="content/ds083.2" /-->

      <featureCollection name="LOCAL_FNL"
                         path="LOCALFNL"
                         harvest="true"
                         featureType="GRIB">


        <metadata inherited="true">
    <serviceName>all</serviceName>
          <documentation type="summary">LOCAL FNLS FOR TESING
          </documentation>
          <dataFormat>GRIB-1</dataFormat>
        </metadata>

        <collection
    spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$"
            timePartition="1 day"
            dateFormatMark="#fnl_#yyyyMMdd_HH"

            />

        <update startup="true" rescan="0 0 0 * * ? *"/>
        <gribConfig datasetTypes="Best Files"/>

      </featureCollection>
    </catalog>

    Upon restarting tomcat, the directory was scanned and I was met
    with numerous "Too many open files" errors:

    2013-04-29T13:12:17.795 -0600 [     36341][        ] ERROR -
    fc.LOCAL_FNL - Grib1CollectionBuilder LOCAL_FNL-20130416 :
    reading/Creating gbx9 index for file
    /glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c
    failed
    java.io.FileNotFoundException:
    
/usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9
    (Too many open files)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
        at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199)
        at
    ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
        at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
        at
    
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)


        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
        at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
        at
    thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
        at
    thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
        at
    
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
    
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
    
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
    ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
        at
    
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
        at
    
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
        at
    
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at
    
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
    2013-04-29T13:12:17.796 -0600 [     36342][        ] ERROR -
    fc.LOCAL_FNL -  Failed to open partition LOCAL_FNL-20130416
    java.io.FileNotFoundException:
    
/usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx
    (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
        at
    ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
        at
    ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
        at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
        at
    
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)


        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
        at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
        at
    thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
        at
    thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
        at
    
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
    
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
    
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
    ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
        at
    
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
        at
    
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
        at
    
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at
    
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
    2013-04-29T13:12:17.803 -0600 [     36349][        ] ERROR -
    fc.LOCAL_FNL - Fail to create/update collection
    java.io.FileNotFoundException:
    /glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx
    (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
        at
    ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
        at
    ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163)
        at
    
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
        at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
        at
    
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)


        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
        at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
        at
    thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
        at
    thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
        at
    
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
    
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
    
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
    ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
        at
    
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
        at
    
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
        at
    
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at
    
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)


    So I pared back the dataset from 3.5 years worth of 4 daily files
    to just the first month (still 4 files daily) of 2010-2013.  This
    eliminated the "Too many open files" errors, and created the
    respective *.gbx9 and .ncx files, but threw a cast error:

    2013-04-29T13:40:04.188 -0600 [    108434][        ] INFO -
    fc.LOCAL_FNL - fnl_20130131_00_00_c: createIndex
    
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx
    2013-04-29T13:40:04.351 -0600 [    108597][        ] INFO -
    fc.LOCAL_FNL - fnl_20130131_06_00_c: createIndex
    
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx
    2013-04-29T13:40:04.508 -0600 [    108754][        ] INFO -
    fc.LOCAL_FNL - fnl_20130131_12_00_c: createIndex
    
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx
    2013-04-29T13:40:04.649 -0600 [    108895][        ] INFO -
    fc.LOCAL_FNL - fnl_20130131_18_00_c: createIndex
    
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx
    2013-04-29T13:40:06.936 -0600 [    111182][        ] ERROR -
    fc.LOCAL_FNL - Fail to create/update collection
    java.lang.ClassCastException:
    ucar.nc2.grib.GribCollection$VariableIndex cannot be cast to
    ucar.nc2.grib.TimePartition$VariableIndexPartitioned
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197)
        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)


        at
    
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
        at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
        at
    thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
        at
    thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
        at
    
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
    
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
    
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
    ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
        at
    
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
        at
    
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
        at
    
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at
    
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)



    And when trying to access my TDS from a browser, the catalog could
    not be made:


    2013-04-29T13:40:33.330 -0600 [    137576][       7] INFO -
    threddsServlet - Remote host: 128.117.8.232 - Request: "GET
    /thredds/catalog/LOCALFNL/catalog.html HTTP/1.1"
    2013-04-29T13:40:33.335 -0600 [    137581][       7] ERROR -
    fc.LOCAL_FNL - Error making catalog for LOCALFNL
    java.lang.NullPointerException
        at
    thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453)
        at
    thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377)
        at
    
thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467)
        at
    thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430)

        at
    
thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
        at
    
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
        at
    
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
        at
    
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
        at
    
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
        at
    
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
        ...


    For grins, I tried changing the partition to

            timePartition="directory"
            dateFormatMark="#grib1/#yyyy"

    And got the same ClassCastException.


    If I'm missing something obvious, please let me know.  I
    understand that the GRIB feature collection is still in flux, so
    please consider this some testing info.  I'm able to make a GRIB
    collection without the time partition, but for file browsing, that
    partition would be a feature we would really want to incorporate.

    Cheers!


    -kevin.

-- Kevin Manross
    NCAR/CISL/Data Support Section
    Phone: (303)-497-1218 <tel:%28303%29-497-1218>
    Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
    Web:http://rda.ucar.edu

    _______________________________________________
    thredds mailing list
    thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    http://www.unidata.ucar.edu/mailing_lists/




--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: Dan.Swank@xxxxxxxx <mailto:Dan.Swank@xxxxxxxx>
Phone: 828-271-4007
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: