Mark Rehbein wrote:
Hi John,
The idea of NcML sounds perfect and will give it a go when you announce
it in the coming weeks.
Just finished new NcML pages at
http://www.unidata.ucar.edu/packages/netcdf/ncml/
We are now experiencing performance issues with reading large portions of
data (2.5million pixels) from our netcdf files. It takes about 7 seconds
to read the data from the netcdf file. The resulting image is 1900 x 1300
pixels, which is full resolution. However, full resolution is rarely
required for requests that cover a large spatial area. So it would be
useful for me if there was a read method on VariableDS that could
subsample the data to get a lower resolution. Hopefully, this would result
in a quicker read time. I am keen to help make the API as fast as
possible, so just let me know how I can help.
1.we have prototyped new version using NIO and are seeing speedups from
2.5 to 8 over current java. a few slides are at
http://www.unidata.ucar.edu/staff/caron/presentations/CurrentWork/CurrentWork.htm
that will take a few months to become operational, but let me know if
you want to alpha test it.
2. the subsampling is possible, but im not so sure it would speed things
up that much when reading on the server. Once you have an array, you can
decimate using
public Array section( Range[] ranges);
(note Range can have a "step" in the constructor).
i will think more about it.
I would like to say, however, that I am finding your API extremely useful
so keep up the good work.
Thanks for your time,
Mark
thanks!
On Mon, 26 May 2003, John Caron wrote:
Hi Mark:
We are working on a thing called "NcML Dataset" that allows you to
describe a dataset that consists of multiple netcdf files using an XML
dialect called NcML. Im trying to get this API finalized also, and will
announce it in the next few weeks so you can have a look, and try it
out.
There may be some performance issues with very large datasets like you
indicate above.I look forward to hearing your thoughts on what is
needed, and perhaps we can work together on making it as fast as
possible.
Currently we are only planning on continuing to provide an API that
extracts rectangular subsets, not diagonals.
Mark Rehbein wrote:
Hi John,
I have made some excellent progress in our satellite imagery software
which is using the netcdf java API you are actively developing.
Our application is a web server based where images/maps are generated on
the web server (tomcat) using the ESRI MapObjects Java Edition API, JAI
and the NetCDF API. The generated maps are then sent to the clients web
browser as a JPEG or PNG. The generated maps have GIS data/layers laid
over the sattelite imagery retrieved from the netCDF file.
I would like to know if you are planning to include in the netcdf API, a
set of classes that could manage multiple netcdf files such that it would
appear to clients of the classes that they were dealing with a single
very large netcdf file. I have a need for this functionality as our
imagery is currently taking up 2GB for each 6 months of imagery and we
plan to host 15 years worth and as such we would like to split our imagery
into netcdf files with 1 months worth of data in each. It would be
expected that the system be able to retrieve a time series that would
span multiple months/years. I will write my own set of classes to handle
the multiple netcdfs, but it might be something that could be included in
the netcdf java API in the future. Please let me know your comments on
this.
Also, I require functionality in the API that could read data from an
image in the netcdf file as show in the following example:
Data along a "diagonal" line from Point 1 (lat:-10,lon:149) to Point 2
(lat:-15,lon:156).
At the moment, I plan to do this by reading all the data bounded by the
above coordinates, using JAI to rotate the image (while applying
pixel interpolation) such that the line between Point 1 and Point 2 is
normal to the Y axis of the rotated image. Then I can read the data along
the line easily. I may have missed something in the doco/manual, but I
could not see anything in the API that could do this for me.
Thanks for all your help so far as I understand that you are busy
stabilising the API in preparation for production release.
cheers
Mark
On Fri, 16 May 2003, John Caron wrote:
Mark Rehbein wrote:
Hi John,
I have noticed that you are actively working on the API as I have seen the
javadoc change on GridCoordSys in the last few hours. I have since
downloaded the library again, as I was using this class for my program.
yes, Im trying to get a stable version out. sorry for the changes.
I have made some progress but have a question and wish to report a
bug/problem.
My question is:
How do I easily convert a java.util.Date to an array index on the time
axis? There is a method GridCoordSys.findXYCoordElement(double, double,
int[]) which handles converting latitude and longitude to array indexes
and I think it would be useful to have an equivalent for the time axis.
how about:
/**
* Given a Date, find the corresponding time index on the time
coordinate axis.
* Can only call this is hasDate() is true.
* This will return
* <ul>
* <li> i, if time(i) <= d < time(i+1).
* <li> -1, if d < time(0)
* <li> n-1, if d > time(n-1), where n is length of time coordinates
* </ul>
*
* @param d date to look for
* @param corresponding time index on the time coordinate axis
* @throws UnsupportedOperationException is no time axis or isDate() false
*/
public int findTimeCoordElement(Date d) ;
The bug or problem that I have discovered is a negative seek offset error
as shown below:
<error>
java.io.IOException: Negative seek offset
at java.io.RandomAccessFile.seek(Native Method)
at ucar.netcdf.RandomAccessFile.read_(RandomAccessFile.java:508)
at ucar.netcdf.RandomAccessFile.seek(RandomAccessFile.java:350)
at
ucar.netcdf.NetcdfFile$V1FloatIo.readArray(NetcdfFile.java:1447)
at ucar.netcdf.NetcdfFile$V1Io.copyout(NetcdfFile.java:896)
at ucar.netcdf.Variable.copyout(Variable.java:276)
at ucar.nc2.Variable.read(Variable.java:184)
at ucar.nc2.dataset.VariableDS.read(VariableDS.java:334)
at
aims.app.science.biodiv.sstAtlas.ImageLibrary.readData(ImageLibrary.java:166)
at
aims.app.science.biodiv.sstAtlas.ImageLibrary.main(ImageLibrary.java:423)
done!
</error>
I have appended the method I have written below for your convenience, if
needed. The method works fine when I use:
latitude= -7.0 and longitude= 157.99
latitude= -27.0 and longitude= 157.99
latitude= -27.0 and longitude= 142.00
latitude= -7.0 and longitude= 142.00
but it throws the above error when using lat= -19.6 and lon= 150.5
which are coordinates within the bounds of coordinates which work!
i assume you found this problem
I have noticed that the *bottom* *left* corner of my images (in lat,lon
coordinates) represents 0,0 in array index coordinates. I would have
expected that the *top* *left* corner of my images (in lat,lon
coordinates) would be represented by 0,0 in array index coordinates.
latitudes are stored both ways: north to south ans south to north.
I am going to add a routine to normalize this in GeoGrid, havent
figured out the API yet.
