Hi Dan,
Thank you (and Tim Pugh) for the reminder to look at the JVM settings on
the OS. Through John and Lansings's work, I have been able to get some
of the time partitioning to work with my data set.
I am still getting the "too many files" error when trying to set up a
daily partition, but no such errors with a monthly partition. I have
yet to modify the OS JVM, and I'll try that today.
Much appreciated!
-kevin.
On 5/1/13 7:49 AM, Dan Swank - NOAA Affiliate wrote:
Just a quick plug...
Been closely working with Jahn Caron (up until he left for EGU), and
trying to get featureCollections working for all the major NCDC-NOMADS
Archive datasets. Both Grib1 & 2 ~ With moderate success.
Alot (but not all) of the issues I found were due to problems on our
own file system. Not permissions, but rather
zero-byte and/or corrupt/damaged GRIB records wrecking havoc
with the GRIB scanner and cause it to fail to create the master
time partition / index files.
Caron added some modifications in the last few .wars to make
the GRIB scanner more tolerant, but as you mentioned there are still
lingering issues with it.
The "too many open files" sounds like you may need to adjust
an OS or JAVA env setting. Not an expert on this and perhaps
someone else can chime in. In short, it may not necessarily be
related to the featureCollectionScan.
-Dan
NCDC-NOMADS project
On Mon, Apr 29, 2013 at 4:17 PM, Kevin Manross <manross@xxxxxxxx
<mailto:manross@xxxxxxxx>> wrote:
Here's an update on attempting a Time Partition for model analyses
that are in GRIB1:
I have created a local data structure for testing where I can
write to the file directories (until I can get permission to write
to the file directories of our main archive). The structure is as
follows:
/glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c
When I found the message thread :
http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html
I took the advice mentioned therein and grabbed the 4.3.17 version
of the thredds war, and set my catalog.xml config as follows:
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change
this to fit your server!"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
<http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0>
xmlns:xlink="http://www.w3.org/1999/xlink";
<http://www.w3.org/1999/xlink>>
<service name="all" base="" serviceType="compound">
<service name="odap" serviceType="OpenDAP"
base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer"
base="/thredds/fileServer/" />
<!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
<service name="wms" serviceType="WMS" base="/thredds/wms/" />
<service name="ncss" serviceType="NetcdfSubset"
base="/thredds/ncss/grid/" />
</service>
<!--datasetRoot path="FNL" location="content/ds083.2" /-->
<featureCollection name="LOCAL_FNL"
path="LOCALFNL"
harvest="true"
featureType="GRIB">
<metadata inherited="true">
<serviceName>all</serviceName>
<documentation type="summary">LOCAL FNLS FOR TESING
</documentation>
<dataFormat>GRIB-1</dataFormat>
</metadata>
<collection
spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$"
timePartition="1 day"
dateFormatMark="#fnl_#yyyyMMdd_HH"
/>
<update startup="true" rescan="0 0 0 * * ? *"/>
<gribConfig datasetTypes="Best Files"/>
</featureCollection>
</catalog>
Upon restarting tomcat, the directory was scanned and I was met
with numerous "Too many open files" errors:
2013-04-29T13:12:17.795 -0600 [ 36341][ ] ERROR -
fc.LOCAL_FNL - Grib1CollectionBuilder LOCAL_FNL-20130416 :
reading/Creating gbx9 index for file
/glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c
failed
java.io.FileNotFoundException:
/usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9
(Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199)
at
ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at
thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
at
thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
at
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
at
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2013-04-29T13:12:17.796 -0600 [ 36342][ ] ERROR -
fc.LOCAL_FNL - Failed to open partition LOCAL_FNL-20130416
java.io.FileNotFoundException:
/usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
at
ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
at
ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at
thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
at
thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
at
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
at
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2013-04-29T13:12:17.803 -0600 [ 36349][ ] ERROR -
fc.LOCAL_FNL - Fail to create/update collection
java.io.FileNotFoundException:
/glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
at
ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
at
ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163)
at
ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85)
at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at
ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at
thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
at
thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
at
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
at
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
So I pared back the dataset from 3.5 years worth of 4 daily files
to just the first month (still 4 files daily) of 2010-2013. This
eliminated the "Too many open files" errors, and created the
respective *.gbx9 and .ncx files, but threw a cast error:
2013-04-29T13:40:04.188 -0600 [ 108434][ ] INFO -
fc.LOCAL_FNL - fnl_20130131_00_00_c: createIndex
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx
2013-04-29T13:40:04.351 -0600 [ 108597][ ] INFO -
fc.LOCAL_FNL - fnl_20130131_06_00_c: createIndex
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx
2013-04-29T13:40:04.508 -0600 [ 108754][ ] INFO -
fc.LOCAL_FNL - fnl_20130131_12_00_c: createIndex
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx
2013-04-29T13:40:04.649 -0600 [ 108895][ ] INFO -
fc.LOCAL_FNL - fnl_20130131_18_00_c: createIndex
/glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx
2013-04-29T13:40:06.936 -0600 [ 111182][ ] ERROR -
fc.LOCAL_FNL - Fail to create/update collection
java.lang.ClassCastException:
ucar.nc2.grib.GribCollection$VariableIndex cannot be cast to
ucar.nc2.grib.TimePartition$VariableIndexPartitioned
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128)
at
ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at
thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224)
at
thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at
thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at
thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213)
at
thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392)
at
thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
And when trying to access my TDS from a browser, the catalog could
not be made:
2013-04-29T13:40:33.330 -0600 [ 137576][ 7] INFO -
threddsServlet - Remote host: 128.117.8.232 - Request: "GET
/thredds/catalog/LOCALFNL/catalog.html HTTP/1.1"
2013-04-29T13:40:33.335 -0600 [ 137581][ 7] ERROR -
fc.LOCAL_FNL - Error making catalog for LOCALFNL
java.lang.NullPointerException
at
thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453)
at
thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377)
at
thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467)
at
thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430)
at
thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
...
For grins, I tried changing the partition to
timePartition="directory"
dateFormatMark="#grib1/#yyyy"
And got the same ClassCastException.
If I'm missing something obvious, please let me know. I
understand that the GRIB feature collection is still in flux, so
please consider this some testing info. I'm able to make a GRIB
collection without the time partition, but for file browsing, that
partition would be a feature we would really want to incorporate.
Cheers!
-kevin.
--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218 <tel:%28303%29-497-1218>
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: Dan.Swank@xxxxxxxx <mailto:Dan.Swank@xxxxxxxx>
Phone: 828-271-4007
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================
--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
