I have to work with a lot of data, which are structured differently (in
naming and location conventions)
from a product to another one. I wonder if all of these would be easilly
configurable or if we have to
change their organisation. So I have a few questions:
- I would like to know more about all the scan fonctionnalities:
Are they working the same way for matching regExp, and is the full file
path tested by this regExp or only the file name ?
If the entire path is tested, does dateFormatMark, runDateMatcher and
other date markers use the full path to ?
In other word, is it possible to extract date informations from
directory name as from file name ?
- In which cases netcdf files are opened when building catalogs, and if I
provide needed metadata,
will they be opened to (optimizations)?
- I have a problem with the following datasetFmrc:
Dataset definition:
-------------------
<datasetFmrc name="TestDatasetFmrc" path="testDatasetFmrc">
<metadata inherited="true">
<serviceName>multiService</serviceName>
<dataType>Grid</dataType>
<dataFormat>NetCDF</dataFormat>
</metadata>
<netcdf xmlns="
http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; enhance="true">
<aggregation dimName="run" type="forecastModelRunCollection"
recheckEvery="1 hour">
<scan
location="/home/coriolis_exp/spool/co01/co0123/co012302/co01230204/archive/"
regExp="(.*)MM5_200710([0-9]{2})\.nc$"
suffix=".nc" subdirs="true"
dateFormatMark="MM5_#yyyyMMdd"
olderThan="5 min" />
</aggregation>
</netcdf>
</datasetFmrc>
The files are all in the same directory:
----------------------------------------
|-- /home/coriolis_exp/spool/co01/co0123/co012302/co01230204/archive/
|-- ...
|-- MM5_20071008.nc
|-- MM5_20071009.nc
|-- ...
These files are structured as below (ncdump -h):
------------------------------------------------
netcdf MM5_20071009 {
dimensions:
lon = 440 ;
lat = 185 ;
time = UNLIMITED ; // (41 currently)
variables:
float lon(lon) ;
lon:long_name = "longitude" ;
lon:units = "degree_east" ;
float lat(lat) ;
lat:long_name = "latitude" ;
lat:units = "degree_north" ;
double time(time) ;
time:long_name = "seconds since the
origin date" ;
time:time_origin = "01-JAN-0000
00:00:00" ;
time:units = "seconds" ;
float uwd(time, lat, lon) ;
uwd:long_name = "10-meter U Component" ;
uwd:units = "m/s" ;
float vwd(time, lat, lon) ;
vwd:long_name = "10-meter V Component" ;
vwd:units = "m/s" ;
float pmer(time, lat, lon) ;
pmer:long_name = "Sea level pressure" ;
pmer:units = "Pa" ;
float rad(time, lat, lon) ;
rad:long_name = "solar radiation flux" ;
rad:units = "w/m^2" ;
float sat(time, lat, lon) ;
sat:long_name = "2-meter Temperature" ;
sat:units = "C degrees" ;
float rain(time, lat, lon) ;
rain:long_name = "precipitation during
one period" ;
rain:units = "kg/m^2" ;
float hflw(time, lat, lon) ;
hflw:long_name = "long wave heat flux" ;
hflw:units = "w/m^2" ;
float rh(time, lat, lon) ;
rh:long_name = "Relative humidity" ;
rh:units = "%" ;
// global attributes:
:Conventions = "COARDS" ;
:host = "ftp.unidata.ucar.edu" ;
:directory = "pub/netcdf/Conventions/COARDS" ;
:title = "MM5 version 3 format output on sigma levels" ;
:history = "Tue Oct 9 04:46:11 2007: ncrename -d
j_cross,lon final.nc\n",
"Tue Oct 9 04:46:11 2007: ncrename -d
i_cross,lat final.nc\n",
"Tue Oct 9 04:46:11 2007: ncatted -O -a
history,global,o,c, final.nc\n",
"" ;
:institution = "CETEMPS - University of L\'Aquila,
Italy" ;
:source = "NCAR MM5 model version 3" ;
:references = "http://www.mmm.ucar.edu/mm5/mm5-home.html";
;
:comment = "No comments yet" ;
I can access the dataset with the web browser:
----------------------------------------------
TestDatasetFmrc
Forecast Model Run Collection (2D time coordinates)
Best Time Series
Forecast Model Run
Constant Forecast Offset
Constant Forecast Date
But when I try to get any of the proposed datasets, it results in:
------------------------------------------------------------------
"Etat HTTP 400 - NULL message java.lang.NullPointerException" in the
browser
And in the threddsServlet.log:
------------------------------
2007-10-09T14:31:42.373 +0200 [ 3177637][ 45] ERROR -
thredds.servlet.ServletUtil - handleException
java.lang.NullPointerException
at thredds.servlet.ViewServlet$IDV.getViewerLinkHtml(ViewServlet.java
:212)
at thredds.servlet.ViewServlet.showViewers(ViewServlet.java:159)
at thredds.servlet.CatalogServicesServlet.showDataset(
CatalogServicesServlet.java:324)
at thredds.servlet.CatalogServicesServlet.handleCatalogServiceRequest(
CatalogServicesServlet.java:201)
at thredds.servlet.CatalogServicesServlet.doGet(
CatalogServicesServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:178)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
AuthenticatorBase.java:432)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
I can't see what's going wrong...
Thanks for your help.
Jonathan
