[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #OZO-122681]: changing variable shape with NcML

> I saw a teaser for "logicalView" in the ncml schema. What is the thought
> there?

yes, its a teaser. i started, but havent finished it.

> 
> Plan B:
> I've decided to run with a custom IOSP. I have a bit of a chicken and
> egg problem. I'm using NcML because my data files (plain binary) have
> absolutely no metadata. However, I need some metadata for the IOSP to
> populate the NetcdfFile. 

if you have a proprietary binary, you need an IOSP. But IOSP doesnt have access 
to the NcML, as youve discovered. But if IOSP doesnt need the metaddata, you 
can add it after the IOSP does its job.

Or is this still the HDF5 reshape problem?

> A simple 1D array of unknown length will work,
> but I'm having issues with the Dimension. Note, I am using netcdf java
> version 4 which has dropped Dimension.UNKNOWN. I almost got things
> working with Dimension("t", Dimension.VLEN). In the NcML I define a
> "time" dimension of fixed length and defined the data variable with that
> dimension. When I try to read the data, I end up back in our old friend
> ucar.ma2.Section.fill():
> 
> ucar.ma2.InvalidRangeException: Illegal Range for dimension 0: last
> requested 9 > max -2
> at ucar.ma2.Section.fill(Section.java:143)
> at ucar.nc2.Variable.read(Variable.java:776)
> at ucar.nc2.dataset.VariableDS._read(VariableDS.java:496)
> at ucar.nc2.Variable.read(Variable.java:776)
> at ucar.nc2.Variable.read(Variable.java:722)
> at lasp.tss.TimeSeriesDataset.readData(TimeSeriesDataset.java:577)
> 
> The Variable has the shape: [-1] from the unknown length Dimension that
> I used. The variable that I defined in the ncml with the fixed length
> Dimension is being circumvented by delegating to the orgVar again. Very
> similar issue to my original reshaping one.

the VLEN Dimension can only be used for sequences, and the processing of it is 
still, um, immature.

If youre just trying to pass aux data to your IOSP, this is too complex.



> 
> Plan C:
> I see that the NcML spec supports "iospParam". I figured I could use
> this to pass in dimension sizes so I could define my original NetcdfFile
> with fixed length Dimensions in the IOSP. The docs
> (http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/AnnotatedSchema4.html)
> state that "the optional iospParam is passed to the IOSP through the
> IOServiceProvider.setSpecial() method." I see no such creature nor a way
> to access the parameters. Does the "iospParam" feature work?

yes, it will be passed to your IOSP through 

  public Object IOSPServiceProvider.sendIospMessage( Object message);

see NetcdfFile line 716

So you can pass an arbitrary String of data to the IOSP by using the iospParam 
in NcML. Perhaps this will work for you?

> 
> Plan D:
> Use the NcML file for the IOSP to get it's metadata so it can make the
> NetcdfFile so the NcML can serve the data via the IOSP. That's sounding
> a bit too circuitous.
> 
> Plan E:
> Get the data provider to put metadata in the data files. Or in a sister
> file. Which might as well be the ncml file (see Plan D). Aaaagh.
> 
> By the way, I recently downloaded the netcdf-java 4 source zip file
> (ftp://ftp.unidata.ucar.edu/pub/netcdf-java/v4.0/ncSrc-4.0.zip) and
> tried to use it in eclipse to see if I might have missed a bug fix. It
> is a few times larger than the previous copy I had and this one has
> similar issues that I had some time ago with the XMLConstants. I'm
> guessing that it's the old stax-api-1.0.1.jar that you include winning
> in the classpath. Did some inadvertent things make it into the source
> release this time?
> 
> Thanks,
> Doug
> 
> Unidata netCDF Java Support wrote:
> > Hi Doug:
> >
> > I though about but never implemented that feature in NcML. Ill give it some 
> > more thought to see how easy it would be. You could maybe do it yourself by 
> > doing:
> >
> >   Array orgData = orgVar.read();
> >   Array newData = orgData.reshape( myShape);
> >   return newData.section( mySection);
> >
> > note all data is always read in. assuming you have enough memory, you could 
> > cache the data with
> >
> >   orgVar.setCache( orgData);
> >
> > so subsequent reads would get it.
> >
> >
> >> I've hit a roadblock and haven't had any luck on the mail list. Is NcML
> >> designed to be able to change the rank of an original dataset?
> >>
> >> ---
> >> I'm using relatively new version 4 NetCDF Java code.
> >> I have an HDF5 file with a 1D array of data, but I secretly know that it
> >> represents a 2D array. I was hoping that I could simply get away with
> >> defining a 2D shape for the variable in NcML:
> >>
> >> <dimension name="time" length="149016" isUnlimited="true" />
> >> <dimension name="vector_dimension" length="2" />
> >> <variable name = "B" orgName="ExtendibleArray" shape="time
> >> vector_dimension" type="double">
> >>
> >> I was getting away with it till I tried to subset the Variable with
> >> read(int[] origin, int[] shape). When it got to VariableDS._read(Section
> >> section), the code delegated to the 1D "orgVar" with the 2D section
> >> which made it sad:
> >>
> >> ucar.ma2.InvalidRangeException: Number of ranges in section (2) must be = 1
> >> at ucar.ma2.Section.fill(Section.java:143)
> >> at ucar.nc2.Variable.read(Variable.java:776)
> >> at ucar.nc2.dataset.VariableDS._read(VariableDS.java:496)
> >> at ucar.nc2.Variable.read(Variable.java:776)
> >> at ucar.nc2.Variable.read(Variable.java:722)
> >>
> >> Was I just being overly optimistic in trying to change the shape/rank
> >> this way? Any suggestions on how to get this as a 2D array via the NcML
> >> Reader?
> >>
> >> Thanks,
> >> Doug
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: OZO-122681
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Closed
> >
> 
> 


Ticket Details
===================
Ticket ID: OZO-122681
Department: Support netCDF Java
Priority: Normal
Status: Open