[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #ARF-381071]: Thredds + scanning (fmrc) with unreadable subdirectories

Hi Mike,

Thanks for the report. I found the same problem in our 4.0 development code as 
well. Even though the underlying scan abstraction is different, we still 
weren't catching this problem.

I'll get a fix in the next 3.16 release that will provide better error messages 
at least and hopefully drop unreadable directories completely. But it looks 
like it may depend on the platform the TDS is running on. I need to do some 
more testing but on Windows in some situations java.io.File.canRead() doesn't 
give the result I would expect ("false") when called on an unreadable directory.

I'll work on this more tomorrow and let you know what I find.

Thanks again,

Ethan

> This began as a support request, but I've subsequently found the problem
> and fixed it.  However, it was sufficiently obscure that I felt it was
> worth emailing you to ensure it goes into the searchable record (or gets
> fixed).
> 
> I'd been having a great deal of difficulty scanning a directory of
> model-run netcdfs (using the FMRC configuration in catalog.xml).  I
> could individually specify netcdfs and my configuration would work, but
> every combination I tried with scans would fail with an error like:
> 
> 2008-09-08T17:39:07.185 +0100 [     70353][      14] ERROR -
> thredds.server.opendap.NcDODSServlet - Error
> java.lang.NullPointerException
> at
> ucar.nc2.ncml.Aggregation$DirectoryScan.scanDirectory(Aggregation.java:928)
> at
> ucar.nc2.ncml.Aggregation$DirectoryScan.scanDirectory(Aggregation.java:933)
> at
> ucar.nc2.ncml.Aggregation$DirectoryScan.scanDirectory(Aggregation.java:917)
> at ucar.nc2.ncml.Aggregation.scan(Aggregation.java:803)
> at ucar.nc2.ncml.Aggregation.finish(Aggregation.java:286)
> at ucar.nc2.ncml.NcMLReader.readNetcdf(NcMLReader.java:397)
> 
> The basic configuration was:
> ------------------
> <datasetFmrc name="Met Office MRCS scan"
> path="metoffice_mrcs_aggscan" collectionType="ForecastModelRuns">
> <serviceName>opendap_service</serviceName>
> <netcdf
> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";
> enhance="true">
> <aggregation dimName="runtime" type="forecastModelRunCollection"
> timeUnitsChange="true">
> <scan location="/ncof/" subdirs="false" suffix=".nc"
> dateFormatMark="MRCS_biology_#ddMMyy" enhance="true"/>
> </aggregation>
> </netcdf>
> </datasetFmrc>
> ------------------
> and the system is RHEL4 Linux, running Tomcat 5.5.23 and THREDDS 3.16.39
> (build 2008-06-27 22:55:54)
> 
> To cut a long story short, the problem turned out to be that I had a
> non-readable subdirectory (lost+found, but any directory has the same
> effect) in the netcdf directory.  Even with subdirs="false", this still
> causes the catalog generation to fail.  As soon as the directory
> permissions are adjusted (+rx) or the directory removed, the catalog
> generation works perfectly.
> 
> You may wish to look into the code that does the scans, such that it at
> least gives a useful error message in the logs when this occurs or,
> ideally, ignores and warns about unreadable subdirs only if the subdirs
> option is set to true :)
> 
> Apart from that frustration, I've generally been very impressed with
> thredds - keep up the good work :)
> 
> Cheers,
> 
> Mike Grant.
> Plymouth Marine Laboratory, UK.



Ticket Details
===================
Ticket ID: ARF-381071
Department: Support THREDDS
Priority: Normal
Status: Open