Re: [netcdf-java] IOSP open file risk

  • Subject: Re: [netcdf-java] IOSP open file risk
  • From: John Caron <caron@xxxxxxxxxxxxxxxx>
  • Date: Sun, 06 Dec 2009 06:41:00 -0700
Doug Lindholm wrote:
I have my own abstract iosp that extends AbstractIOServiceProvider. It has a single "open" method that works for all my IOSPs. It calls an "init" method so subclasses have a hook to set up any resources like database connections. It does nothing with the RandomAccessFile (because most of my data files have no metadata or aren't files at all). I end up using the ncml file itself to get the metadata to make the Dimensions and Variables to add to the NetcdfFile that I get handed in "open". I'd like to think that NetCDF could define the NetcdfFile for me based solely on the ncml metadata (not including the "location" data and whatever metadata it can find there). If I don't like what it gives me, I can overwrite "open" and clean it up.

On a related note, I would like to have control of the "location" when the ncml specifies a custom iosp. I might want to use a special URI such as a jdbc connection URL. Right now, I set location to "/dev/null" (NetCDF complains if the location doesn't exist) and encode connection info in the iospParam. I'd rather not be limited to data "locations" that can only be represented as a RandomAccessFile.

"Fixes" for these would remove a significant hurdle in an API that otherwise is performing miracles for me. I might be enticed to contribute to the effort.

Thanks,
Doug


I think last time we discussed this, I wrote:

"The problem is that a location doesnt specify the IOSP needed to handle it. We have to 
ask all the IOSPs we know about to find out who handles it. The IOSP needs to examine the 
contents of the dataset to decide if "its mine". Rather than having each IOSP open 
the file, we pass each the RandomAccessFile, for efficiency. The IOSP design is obviously 
oriented to using RandomAccessFile.

"Now an NcML file can specify the IOSP, so potentially this might be a 
workaround this problem. The presumed use of NcML is to wrap an existing NetcdfFile. 
You want to reverse that and let your IOSP's NetcdfFile wrap the NcML, by giving it 
access to the NcML.

"I could see trying an experimental version that passes the NcML to the iosp 
specified in the NcML.  I would guess that NcMLReader would not further wrap it, 
though maybe we can relax that later. Im not sure what issues we'd find, and but we 
could try it, though i wouldnt commit to supporting it until we have some experience.

"The question is whether an IOSP is the right level to implement your database 
access. Alternatively, one could subclass NetcdfFile, which is what other non-file 
implementations like opendap, thredds, and cdmremote do. The problem is that theres 
no mechanism to associate the location with a factory that creates those subclasses. 
I suppose we could create one, though.

"So have you considered subclassing NetcdfFile instead of writing an IOSP ?
I never heard back from you, what are your thoughts?

In any case, it wont be possible to allow an iosp to redefine the meaning of "location". 
If you dont have a "referenced dataset", then that must be left blank.



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: