HI Sean,
Thanks a lot for help me! And bad news, it is still there.
Sorry about forgetting to mention before, I have already been using tdm 4.6.5
the link just like you provided.
using:
/usr/java -Xmx4g -Dtds.content.root.path="/usr/local/tomcat/content" -jar
tdmFat-4.6.5.jar -tds "http://localhost:8080/"; -cred tdm:secret -log DEBUG
I actually use "test" for rewrite and just change to "always" to see whether
anything could happen.
the tdm log is:
2016-04-19T13:09:00.006 +0800 DEBUG - 2016-04-19T05:09:02016-04-19T13:05:51.447
+0800 INFO - FeatureCollection config=FeatureCollectionConfig name ='ncep'
collectionName='ncep_all' type='GRIB2'
spec='/home/jtang/Documents/THREDDS_NEW/**/.*\.grib2$'
timePartition =directory
updateConfig =UpdateConfig{userDefined=false, recheckAfter='null',
rescan='null', triggerOk=true, updateType=never}
tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null', rescan='0 0/1
* * * ? *', triggerOk=true, updateType=test}
ProtoConfig{choice=Penultimate, change='null', param='null',
outerNcml='null', cacheAll=true}
hasInnerNcml =false
GribConfig{datasets=[TwoD, Best, Files, Latest], useGenType=false,
useTableVersion=false, intvMerge=true, useCenter=false,
filesSortIncreasing=true}
2016-04-19T13:05:51.458 +0800 INFO - scheduleJob startup scan force=test for
'ncep_all' at 2016-04-19T13:05:54.453+0800
2016-04-19T13:05:51.467 +0800 INFO - scheduleJob recurring scan for 'ncep_all'
cronExpr=0 0/1 * * * ? *
2016-04-19T13:05:54.494 +0800 DEBUG - Grib2CollectionBuilder
ncep_all-2016041500: makeGroups
2016-04-19T13:05:54.494 +0800 DEBUG -
dcm=thredds.inventory.partition.DirectoryCollection@4c3c53c4
2016-04-19T13:05:54.635 +0800 DEBUG - Index written: f003.grib2.gbx9 == 366
records
2016-04-19T13:05:55.390 +0800 DEBUG - Counter: nvars=135 records 366/366
(1.000000) filtered=0 dups=0 (0.000000)
2016-04-19T13:05:55.415 +0800 DEBUG - createIndex for
/home/jtang/Documents/THREDDS_NEW/2016041500/ncep_all-2016041500.ncx3
2016-04-19T13:05:55.430 +0800 DEBUG - write RecordMaps: bytes = 6322 record =
366 bytesPerRecord=17
2016-04-19T13:05:55.486 +0800 DEBUG - write GribCollectionIndex= 10150 bytes
2016-04-19T13:05:55.487 +0800 DEBUG - That took 994 msecs
2016-04-19T13:05:55.585 +0800 DEBUG - Using canonical partition
ncep_all-2016041500
2016-04-19T13:05:55.662 +0800 INFO - updateGribCollection ncep_all changed
true took 1188 msecs
2016-04-19T13:05:55.662 +0800 DEBUG - 2016-04-19T05:05:55.662Z ncep_all changed
true
2016-04-19T13:06:01.650 +0800 INFO - send trigger to
http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=ncep_all
status = 200
2016-04-19T13:07:00.003 +0800 DEBUG - Grib2CollectionBuilder
ncep_all-2016041500: makeGroups
2016-04-19T13:07:00.003 +0800 DEBUG -
dcm=thredds.inventory.partition.DirectoryCollection@34fedf53
2016-04-19T13:07:00.037 +0800 DEBUG - Index written: f006.grib2.gbx9 == 366
records
2016-04-19T13:07:00.077 +0800 DEBUG - Counter: nvars=135 records 732/732
(1.000000) filtered=0 dups=0 (0.000000)
2016-04-19T13:07:00.077 +0800 DEBUG - createIndex for
/home/jtang/Documents/THREDDS_NEW/2016041500/ncep_all-2016041500.ncx3
2016-04-19T13:07:00.085 +0800 DEBUG - write RecordMaps: bytes = 11061 record
= 732 bytesPerRecord=15
2016-04-19T13:07:00.089 +0800 DEBUG - write GribCollectionIndex= 10195 bytes
2016-04-19T13:07:00.089 +0800 DEBUG - That took 86 msecs
2016-04-19T13:07:00.093 +0800 DEBUG - Using canonical partition
ncep_all-2016041500
2016-04-19T13:07:00.116 +0800 INFO - updateGribCollection ncep_all changed
true took 114 msecs
2016-04-19T13:07:00.117 +0800 DEBUG - 2016-04-19T05:07:00.116Z ncep_all changed
true
2016-04-19T13:07:00.133 +0800 INFO - send trigger to
http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=ncep_all
status = 200
2016-04-19T13:08:00.015 +0800 INFO - updateGribCollection ncep_all changed
false took 14 msecs
2016-04-19T13:08:00.015 +0800 DEBUG - 2016-04-19T05:08:00.015Z ncep_all changed
false
2016-04-19T13:09:00.006 +0800 INFO - updateGribCollection ncep_all changed
false took 5 msecs
2016-04-19T13:09:00.006 +0800 DEBUG - 2016-04-19T05:09:00.006Z ncep_all changed
false
2016-04-19T13:10:00.006 +0800 INFO - updateGribCollection ncep_all changed
false took 4 msecs
2016-04-19T13:10:00.006 +0800 DEBUG - 2016-04-19T05:10:00.006Z ncep_all changed
false
2016-04-19T13:11:00.004 +0800 INFO - updateGribCollection ncep_all changed
false took 3 msecs
2016-04-19T13:11:00.005 +0800 DEBUG - 2016-04-19T05:11:00.005Z ncep_all changed
false
2016-04-19T13:12:00.005 +0800 INFO - updateGribCollection ncep_all changed
false took 4 msecs
0.006Z ncep_all changed false
2016-04-19T13:10:00.006 +0800 INFO - updateGribCollection ncep_all changed
false took 4 msecs
2016-04-19T13:10:00.006 +0800 DEBUG - 2016-04-19T05:10:00.006Z ncep_all changed
false
2016-04-19T13:11:00.004 +0800 INFO - updateGribCollection ncep_all changed
false took 3 msecs
2016-04-19T13:11:00.005 +0800 DEBUG - 2016-04-19T05:11:00.005Z ncep_all changed
false
2016-04-19T13:12:00.005 +0800 INFO - updateGribCollection ncep_all changed
false took 4 msecs
As I described before,
in the http://localhost:8080/thredds/catalog/nwp/NCEP/catalog.html
ncep
|--Full collection
|--Best ncep Time Series
|--Latest Collection for ncep
|--ncep-2016041500
the first two (Full, Best) will be updated as I manually put more file in,
but not for the "Latest" and the "ncep-2016041500".
I know it is very weird and hope you could help get some clue.
Best wishes,
Jian
-----原始邮件-----
发件人:"Sean Arms" <sarms@xxxxxxxx>
发送时间:2016-04-18 23:59:37 (星期一)
收件人: "唐健" <tangjian@xxxxxxxxxx>
抄送: thredds <thredds@xxxxxxxxxxxxxxxx>
主题: Re: [thredds] Cannot get FeatureCollection Updated by tdm
Greetings Jian,
The configuration looks ok, but you could simplify it by removing the update
element. The default for update (if the element is missing) is:
<update startup="nocheck" trigger="allow"/>
Also, I would set the TDM element's rewrite attribute to test, especially if
the collection is really big - otherwise, you may run the risk of not being
able to regenerate the collection index files within your 1 minute window. So,
something like this:
<tdm rewrite="test" rescan="0 0/1 * * * ? *" trigger="allow"/>
Do you kno which version of the TDM you are using? If not, I would go ahead and
download the latest TDM to be sure you are using the most up-to-date version.
http://artifacts.unidata.ucar.edu/content/repositories/unidata-releases/edu/ucar/tdmFat/4.6.5/tdmFat-4.6.5.jar
Let me know if that fixes the issue.
Cheers,
Sean
On Mon, Apr 18, 2016 at 8:17 AM, 唐健 <tangjian@xxxxxxxxxx> wrote:
HI everyone,
I am having hard time upgrading from 4.5.x to 4.6.5.
Everything works fine in 4.5.
The thing is when I am in 4.6.5, using FeatureCollection to get a directory of
Grib2(orGRIB1) files, let's say NCEP_GFS fcst files, the directory is
2016041600, and when the GRIB file comes in, the first time ncx3 and gbx9 was
built, the TDM will send trigger to TDS, and it will be updated. But when more
and more file comes in, it still will send trigger, but the TDS will not
respond to it anymore.
First I thought it could be some problem due to windows or something. So I
install openSuse on my laptop, and install both tomcat8 and Java(JDK)8. And I
get SSL working as the same.
So I have the following XML:
Please help me check if I did something wrong.
in catalog.xml
<catalogRef xlink:title="Global Forecast System (GFS) 0p5deg"
xlink:href="nwp_ncep_gfs_0p5deg.xml" name=""/>
and in nwp_ncep_gfs_0p5deg.xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NMC THREDDS Server Catalog"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
xmlns:xlink="http://www.w3.org/1999/xlink"; version="1.0.2" >
<featureCollection name="ncep" featureType="GRIB2" path="nwp/NCEP" >
<metadata inherited="true">
<serviceName>GribServices</serviceName>
<dataFormat>GRIB-2</dataFormat>
</metadata>
<collection spec="/home/jtang/Documents/THREDDS_NEW/**/.*\.grib2$"
timePartition="directory" name="ncep_all"/>
<update startup="test" trigger="allow"/>
<tdm rewrite="always" rescan="0 0/1 * * * ? *" trigger="allow"/>
</featureCollection>
</catalog>
Since I use "GribServices" as the Service, Full Collection (Reference /
Forecast Time) Dataset, it will change when I manually put more file into the
directory, but the
"http://localhost:8080/thredds/catalog/nwp/NCEP/ncep_all-2016041500/catalog.html";
it will not change.
If anyone have any clue, please feel free to jump in.
Thank you very much.
--
Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA
No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA
No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081
2016-04-19T12:32:08.484 +0800 INFO -
org.springframework.context.support.FileSystemXmlApplicationContext -
Refreshing
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb:
startup date [Tue Apr 19 12:32:08 CST 2016]; root of context hierarchy
2016-04-19T12:32:08.515 +0800 INFO -
org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML
bean definitions from class path resource [resources/application-config.xml]
2016-04-19T12:32:08.730 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Using default implementation for ThreadExecutor
2016-04-19T12:32:08.732 +0800 INFO - org.quartz.simpl.SimpleThreadPool - Job
execution threads will use class loader of thread: main
2016-04-19T12:32:08.743 +0800 INFO - org.quartz.core.SchedulerSignalerImpl -
Initialized Scheduler Signaller of type: class
org.quartz.core.SchedulerSignalerImpl
2016-04-19T12:32:08.744 +0800 INFO - org.quartz.core.QuartzScheduler - Quartz
Scheduler v.2.2.0 created.
2016-04-19T12:32:08.744 +0800 INFO - org.quartz.simpl.RAMJobStore -
RAMJobStore initialized.
2016-04-19T12:32:08.745 +0800 INFO - org.quartz.core.QuartzScheduler -
Scheduler meta-data: Quartz Scheduler (v2.2.0) 'DefaultQuartzScheduler' with
instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support
persistence. and is not clustered.
2016-04-19T12:32:08.745 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource
file in Quartz package: 'quartz.properties'
2016-04-19T12:32:08.745 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Quartz scheduler version: 2.2.0
2016-04-19T12:32:08.745 +0800 INFO - org.quartz.core.QuartzScheduler -
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2016-04-19T12:32:08.892 +0800 INFO - thredds.tdm.Tdm - TDS server added
http://localhost:8080/
2016-04-19T12:32:08.894 +0800 INFO - thredds.tdm.Tdm - ThreddsConfigReader
reading xml file = /usr/local/tomcat/content/thredds/threddsConfig.xml
2016-04-19T12:32:08.924 +0800 INFO - thredds.tdm.Tdm - TDM set
DiskCache2{cachePathPolicy=NestedDirectory, alwaysUseCache=false,
cachePathPolicyParam='null',
root='/usr/local/tomcat/content/thredds/cache/grib/', scourEveryMinutes=-1,
persistMinutes=-1, fail=false}
2016-04-19T12:32:08.957 +0800 INFO - serverStartup - LoggerFactorySpecial add
logger= fc.ncep_all file= /usr/local/tomcat/content/tdm/fc.ncep_all.log
2016-04-19T12:32:08.972 +0800 INFO -
org.springframework.context.support.FileSystemXmlApplicationContext - Closing
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb:
startup date [Tue Apr 19 12:32:08 CST 2016]; root of context hierarchy
2016-04-19T12:32:12.108 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T12:33:00.557 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:05:50.549 +0800 INFO -
org.springframework.context.support.FileSystemXmlApplicationContext -
Refreshing
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb:
startup date [Tue Apr 19 13:05:50 CST 2016]; root of context hierarchy
2016-04-19T13:05:50.608 +0800 INFO -
org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML
bean definitions from class path resource [resources/application-config.xml]
2016-04-19T13:05:51.028 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Using default implementation for ThreadExecutor
2016-04-19T13:05:51.032 +0800 INFO - org.quartz.simpl.SimpleThreadPool - Job
execution threads will use class loader of thread: main
2016-04-19T13:05:51.051 +0800 INFO - org.quartz.core.SchedulerSignalerImpl -
Initialized Scheduler Signaller of type: class
org.quartz.core.SchedulerSignalerImpl
2016-04-19T13:05:51.053 +0800 INFO - org.quartz.core.QuartzScheduler - Quartz
Scheduler v.2.2.0 created.
2016-04-19T13:05:51.054 +0800 INFO - org.quartz.simpl.RAMJobStore -
RAMJobStore initialized.
2016-04-19T13:05:51.056 +0800 INFO - org.quartz.core.QuartzScheduler -
Scheduler meta-data: Quartz Scheduler (v2.2.0) 'DefaultQuartzScheduler' with
instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support
persistence. and is not clustered.
2016-04-19T13:05:51.056 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource
file in Quartz package: 'quartz.properties'
2016-04-19T13:05:51.056 +0800 INFO - org.quartz.impl.StdSchedulerFactory -
Quartz scheduler version: 2.2.0
2016-04-19T13:05:51.056 +0800 INFO - org.quartz.core.QuartzScheduler -
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2016-04-19T13:05:51.354 +0800 INFO - thredds.tdm.Tdm - TDS server added
http://localhost:8080/
2016-04-19T13:05:51.357 +0800 INFO - thredds.tdm.Tdm - ThreddsConfigReader
reading xml file = /usr/local/tomcat/content/thredds/threddsConfig.xml
2016-04-19T13:05:51.406 +0800 INFO - thredds.tdm.Tdm - TDM set
DiskCache2{cachePathPolicy=NestedDirectory, alwaysUseCache=false,
cachePathPolicyParam='null',
root='/usr/local/tomcat/content/thredds/cache/grib/', scourEveryMinutes=-1,
persistMinutes=-1, fail=false}
2016-04-19T13:05:51.445 +0800 INFO - serverStartup - LoggerFactorySpecial add
logger= fc.ncep_all file= /usr/local/tomcat/content/tdm/fc.ncep_all.log
2016-04-19T13:05:51.468 +0800 INFO -
org.springframework.context.support.FileSystemXmlApplicationContext - Closing
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb:
startup date [Tue Apr 19 13:05:50 CST 2016]; root of context hierarchy
2016-04-19T13:05:55.662 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:07:00.116 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:08:00.015 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:09:00.006 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:10:00.006 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:11:00.005 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:12:00.006 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:13:00.004 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:14:00.004 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:15:00.005 +0800 INFO - thredds.tdm.Tdm -
GribCdmIndex.updateGribCollection ncep_all changed false
