Hi,
I have encountered a thredds aggregation problem. I have the very latest
thredds.war (3.16.30), java 1.6 and tomcat 6.0.14. The typicalDataset is
set to penultimate for the aggregation. We are aggregating on the
unlimited variable ocean_time. Our aggregation block in the catalog.xml
file looks like this:
<dataset name="Averages"
ID="roms_mabgom_v4_averages"
urlPath="roms/mabgom/v4/avg" >
<netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="ocean_time" type="joinExisting">
<scan
location="/home/om/dods-data/thredds/roms/mabgom/v4/avg/"
suffix=".nc" />
</aggregation>
</netcdf>
</dataset>
I also looked in the aggregation cache file and all the ncoords and
coordValue fields look good.
The error occurs when you ask for an ocean_time record range that does
not start at the beginning of a file. For example, our nc files have 10
records each so if I ask for ocean_times 0 through 15 or 10 through 25
everything is correct and fine. However, if I ask for 1 through 15 or
even 9 through 15 I get the values for records 10 through 15 then a
bunch of zeros.
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[0:1:15]
Dataset {
Float64 ocean_time[ocean_time = 16];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[16]
4.56116478E9, 4.56120954E9, 4.5612543E9, 4.56129906E9, 4.56134382E9,
4.56138858E9, 4.56143334E9, 4.5614781E9, 4.56152286E9, 4.56156762E9,
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[1:1:15]
Dataset {
Float64 ocean_time[ocean_time = 15];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[15]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[9:1:15]
Dataset {
Float64 ocean_time[ocean_time = 7];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[7]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0
As you can see it's only getting records from the second file. The even
stranger thing is that you get the right data (ubar in this example)
using time records not starting at the beginning of a file.
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ubar[9:1:15][0:1:0][0:1:0]
Dataset {
Float32 ubar[ocean_time = 7][eta_u = 1][xi_u = 1];
} roms/mabgom/v4/avg;
---------------------------------------------
ubar[7][1][1]
[0][0], 0.02386481
[1][0], 0.011283709
[2][0], -0.0020005547
[3][0], -0.02091563
[4][0], -0.036607344
[5][0], -0.04679432
[6][0], -0.05165326
That data for ubar is correct. It's the last record in one file and the
first 6 in another. If you tried to get the time values to line up with
this data it would be wrong:
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[9:1:15],ubar[9:1:15][0:1:0][0:1:0]
Dataset {
Float64 ocean_time[ocean_time = 7];
Float32 ubar[ocean_time = 7][eta_u = 1][xi_u = 1];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[7]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0
ubar[7][1][1]
[0][0], 0.02386481
[1][0], 0.011283709
[2][0], -0.0020005547
[3][0], -0.02091563
[4][0], -0.036607344
[5][0], -0.04679432
[6][0], -0.05165326
That first record of ocean time, you will notice, is the value for time
record 10 (first record in second file) whereas the first entry under
ubar is the value for time record 9 (last record in the first file).
This problem occurs with any aggregation set we have that has files with
more than one record in them. The only reason the problem doesn't occur
with files that have one record is because no matter what record you
start with it's at the beginning of a file.
In logs/catalina.out I get errors like:
2008-02-06T10:33:40.835 -0600 [ 1010687][ 47] ERROR -
ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates
dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0001.nc
If I don't start at the beginning of a file (first time record ending in
0) and/or end at the end of a file. For Example, if I ask for records 5-15:
http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[5:1:15]
produces the catalina.out errors:
2008-02-06T10:50:53.315 -0600 [ 2043167][ 712] ERROR -
ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates
dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0001.nc
2008-02-06T10:50:53.316 -0600 [ 2043168][ 712] ERROR -
ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates
dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0002.nc
As you can see I get one error for not starting that the beginning of a
file and another error for not ending at the end of a file.
However, if I use the "NCDump Variable Data" tool from toolsUI.jar to
access the same range (5-15) no errors are produced in catalina.out and
the correct data values are displayed (no zeros).
Any assistance would be greatly appreciated.
Thanks,
David Robertson
