[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #UXM-196070]: Regarding NCML Aggregating ( montly axis at fixed date )



Greetings Akshay,

So I think I have the ncml fixed to do the right thing. Since each nc file
had time already defined and this was a joinExsiting aggregation, there was
actually no need to do anything with time, like extracting the date/time
from the file name. Please try the attached ncml and see if that works for
you.

Cheers,

Sean


On Sat, Apr 9, 2016 at 9:08 AM, Sean Arms <address@hidden> wrote:

> Oops, I see you did that already. Let me take a look real quick.
>
> Thanks!
>
> Sean
>
> On Sat, Apr 9, 2016 at 9:06 AM, Sean Arms <address@hidden> wrote:
>
>> Greeting Aksha,
>>
>> One thing I see is that the time format is not quite right - what you
>> have is extracting year, minute, day. I'd change yyyymmdd to yyyyMMdd.
>> However, there is something else going on here that isn't quite right.
>> Would it be possible to get a copy of two or three of your files so that I
>> can test locally? If not, the output of  `ncdump -h` for one of them would
>> be helpful.
>>
>> Thanks!
>>
>> Sean
>>
>>
>> address@hidden> wrote:
>>
>>> New Client Reply: Regarding NCML Aggregating  ( montly axis at fixed
>>> date )
>>>
>>> Hi!
>>>
>>> Thanks for reply and for pointing what I am doing wrong, I just modified
>>> ncml file, and executed, I am getting netcdf file as output but though
>>> variable named time is created in netcdf but data its wrong, also timing
>>> information like, months since 1960-01-01 00:00:00 is not added please go
>>> through attachment.
>>>
>>> This is what I did
>>>
>>> akshay@las:~/Desktop$ pwd
>>> /home/akshay/Desktop
>>>
>>> akshay@las:~/Desktop$ ls air/* -1
>>> air/air_19600101.nc
>>> air/air_19600201.nc
>>> air/air_19600301.nc
>>> air/air_19600401.nc
>>> air/air_19600501.nc
>>> air/air_19600601.nc
>>> air/air_19600701.nc
>>> air/air_19600801.nc
>>> air/air_19600901.nc
>>> air/air_19601001.nc
>>> air/air_19601101.nc
>>> air/air_19601201.nc
>>>
>>> akshay@las:~/Desktop$ cat test.ncml
>>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>> <aggregation dimName="TIME" type="joinExisting">
>>>                 <variableAgg name="T"/>
>>>              <scan dateFormatMark="air_#yyyymmdd"
>>> location="/home/akshay/Desktop/air" suffix=".nc" subdirs="false" />
>>> </aggregation>
>>> </netcdf>
>>>
>>> akshay@las:~/Desktop$ java -Xmx1g -classpath netcdfAll-4.6.5.jar
>>> ucar.nc2.dataset.NetcdfDataset -in test.ncml -out test.nc
>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>> further details.
>>> NetcdfDatataset read from test.ncml write to test.nc finished=true
>>>
>>>
>>> akshay@las:~/Desktop$ ncdump -h test.nc
>>> netcdf test {
>>> dimensions:
>>>         LON = 120 ;
>>>         LAT = 115 ;
>>>         bnds = 2 ;
>>>         TIME = 12 ;
>>>         TIME_strlen = 20 ;
>>> variables:
>>>         float LON(LON) ;
>>>                 LON:long_name = "Longitude" ;
>>>                 LON:units = "degrees_east" ;
>>>                 LON:axis = "X" ;
>>>                 LON:modulo = 360.f ;
>>>                 LON:point_spacing = "even" ;
>>>                 LON:standard_name = "longitude" ;
>>>         float LAT(LAT) ;
>>>                 LAT:long_name = "Latitude" ;
>>>                 LAT:units = "degrees_north" ;
>>>                 LAT:axis = "Y" ;
>>>                 LAT:point_spacing = "even" ;
>>>                 LAT:standard_name = "latitude" ;
>>>         double TIME_bnds(TIME, bnds) ;
>>>         float AIR(TIME, LAT, LON) ;
>>>                 AIR:missing_value = 32766.f ;
>>>                 AIR:_FillValue = 32766.f ;
>>>                 AIR:units = "degC" ;
>>>                 AIR:long_name = "Air Temperature Monthly Mean at
>>> Surface" ;
>>>                 AIR:add_offset = 239.65f ;
>>>                 AIR:scale_factor = 0.01f ;
>>>                 AIR:Ferret_Precision_Note = "This variable written using
>>> Ferret was converted from SHORT to FLOAT" ;
>>>                 AIR:history = "From
>>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/coads/1degree/global/enh/air.mean.nc";
>>> ;
>>>         char TIME(TIME, TIME_strlen) ;
>>>                 TIME:_CoordinateAxisType = "Time" ;
>>>                 TIME:long_name = "time coordinate" ;
>>>                 TIME:standard_name = "time" ;
>>>
>>> // global attributes:
>>>                 :history = "FERRET V6.93   1-Apr-16" ;
>>>                 :Conventions = "CF-1.6" ;
>>> }
>>>
>>>
>>> akshay@las:~/Desktop$ ncdump  test.nc | awk '/^ TIME =/{f=1}f'
>>>  TIME =
>>>   "1960-01-01T00:01:00Z",
>>>   "1960-01-01T00:02:00Z",
>>>   "1960-01-01T00:03:00Z",
>>>   "1960-01-01T00:04:00Z",
>>>   "1960-01-01T00:05:00Z",
>>>   "1960-01-01T00:06:00Z",
>>>   "1960-01-01T00:07:00Z",
>>>   "1960-01-01T00:08:00Z",
>>>   "1960-01-01T00:09:00Z",
>>>   "1960-01-01T00:10:00Z",
>>>   "1960-01-01T00:11:00Z",
>>>   "1960-01-01T00:12:00Z" ;
>>> }
>>>
>>>
>>> akshay@las:~/Desktop$ ferret
>>>         NOAA/PMEL TMAP
>>>         FERRET v6.96
>>>         Linux 2.6.32-573.7.1.el6.x86_64 64-bit - 12/02/15
>>>          9-Apr-16 14:36
>>>
>>> yes? use test.nc
>>>  **TMAP ERR: Time axis info missing or incomplete for dataset
>>>              unrecognizable time encoding
>>> yes? sh d
>>>      currently SET data sets:
>>>     1> ./test.nc  (default)
>>>  name     title                             I         J         K
>>>  L         M         N
>>>  TIME_BNDS
>>>                                            1:2       ...       ...
>>>  1:12      ...       ...
>>>  AIR      Air Temperature Monthly Mean at  1:120     1:115     ...
>>>  1:12      ...       ...
>>>  TIME     time coordinate                  ...       ...       ...
>>>  1:12      ...       ...
>>>
>>> yes? sh g air
>>>     GRID GMH2
>>>  name       axis              # pts   start                end
>>>  LON       LONGITUDE          120mr   20.5E                139.5E
>>>  LAT       LATITUDE           115 r   74.5S                39.5N
>>>  normal    Z
>>>  TIME      T                   12 r   1                    12
>>>  normal    E
>>>  normal    F
>>> yes? list time
>>>              VARIABLE : time coordinate
>>>              FILENAME : test.nc
>>>              SUBSET   : 12 points (T)
>>>  1    /  1:"1960-01-01T00:01:00Z"
>>>  2    /  2:"1960-01-01T00:02:00Z"
>>>  3    /  3:"1960-01-01T00:03:00Z"
>>>  4    /  4:"1960-01-01T00:04:00Z"
>>>  5    /  5:"1960-01-01T00:05:00Z"
>>>  6    /  6:"1960-01-01T00:06:00Z"
>>>  7    /  7:"1960-01-01T00:07:00Z"
>>>  8    /  8:"1960-01-01T00:08:00Z"
>>>  9    /  9:"1960-01-01T00:09:00Z"
>>>  10   / 10:"1960-01-01T00:10:00Z"
>>>  11   / 11:"1960-01-01T00:11:00Z"
>>>  12   / 12:"1960-01-01T00:12:00Z"
>>> yes?
>>>
>>>
>>> I expect output like this, above as you can see time index are numeric 1
>>> to 12, and also array time contains datetime stamp but its not correct
>>>
>>> yes? sh g air
>>>     GRID GMH2
>>>  name       axis              # pts   start                end
>>>  LON       LONGITUDE          120mr   20.5E                139.5E
>>>  LAT       LATITUDE           115 r   74.5S                39.5N
>>>  normal    Z
>>>  TIME      T                   12 r   1960-01-01           1960-12-01
>>>  normal    E
>>>  normal    F
>>>
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>> Akshay
>>>
>>>
>>>
>>> ----- Original Message -----
>>> address@hidden>
>>> To: address@hidden
>>> Cc: address@hidden
>>> Sent: Friday, April 8, 2016 10:54:57 PM
>>> Subject: [netCDFJava #UXM-196070]: Regarding NCML Aggregating  ( montly
>>> axis at fixed date )
>>>
>>> Greetings Akshay,
>>>
>>> I think the issue is that you are extracting the date from the filename
>>> (good), but are rewriting the information using the NCML ouside of the
>>> aggregation (i.e. <values start="0" increment="1" />). Can you try using
>>> the same NcML you have, but remove this part of the NcML:
>>>
>>> <variable name="TIME" type="double" shape="TIME">
>>>         <attribute name="units" value="months since 1960-01-01 00:00:00"
>>> />
>>>         <values start="0" increment="1" />
>>>          <attribute name="axis" value="T" />
>>> </variable>
>>>
>>> Thanks!
>>>
>>> Sean
>>>
>>>
>>> address@hidden> wrote:
>>>
>>> > New Ticket: Regarding NCML Aggregating  ( montly axis at fixed date )
>>> >
>>> > Hi!
>>> >
>>> > This query is regarding time variable declaration in NCML, here my aim
>>> > is to define time axis at 1th of every month that is same as that of
>>> > date stamp in filename.
>>> >
>>> >
>>> > My Input :
>>> >
>>> > I have list of files like below
>>> >
>>> > # ls -1 | head -10
>>> > air_19600101.nc
>>> > air_19600201.nc
>>> > air_19600301.nc
>>> > air_19600401.nc
>>> > air_19600501.nc
>>> > air_19600601.nc
>>> > air_19600701.nc
>>> > air_19600801.nc
>>> > air_19600901.nc
>>> > air_19601001.nc
>>> >
>>> >
>>> > This is what I tried :
>>> >
>>> > Following is my ncml
>>> >
>>> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
>>> ">
>>> > <variable name="TIME" type="double" shape="TIME">
>>> >         <attribute name="units" value="months since 1960-01-01
>>> 00:00:00" />
>>> >         <values start="0" increment="1" />
>>> >          <attribute name="axis" value="T" />
>>> > </variable>
>>> > <aggregation dimName="TIME" type="joinExisting">
>>> >              <scan dateFormatMark="air_#yyyymmdd"
>>> location="/path/to/AIR"
>>> > suffix=".nc" subdirs="false" />
>>> > </aggregation>
>>> > </netcdf>
>>> >
>>> >
>>> > This is what I received after restarting thredds data server:
>>> >
>>> > I use ferret for data analysis, this is how time indexes are created
>>> >
>>> > yes? set data "http://127.0.0.1/thredds/dodsC/test/test.nc";
>>> > yes? sh g air
>>> >     GRID GFP1
>>> >  name       axis              # pts   start                end
>>> >  LON       LONGITUDE          120mr   20.5E                139.5E
>>> >  LAT       LATITUDE           115 r   74.5S                39.5N
>>> >  normal    Z
>>> >  TIME      TIME               674 r   01-JAN-1960 12:00    31-JAN-2016
>>> > 12:24
>>> >  normal    E
>>> >  normal    F
>>> > yes? list/l=1:10 t[gt=air]
>>> >              VARIABLE : T
>>> >                         axis TIME
>>> >              FILENAME : test.nc
>>> >              FILEPATH : http://127.0.0.1/thredds/dodsC/test/
>>> >              SUBSET   : 10 points (TIME)
>>> >  01-JAN-1960 /  1:  0.000
>>> >  31-JAN-1960 /  2:  1.000
>>> >  02-MAR-1960 /  3:  2.000
>>> >  01-APR-1960 /  4:  3.000
>>> >  02-MAY-1960 /  5:  4.000
>>> >  01-JUN-1960 /  6:  5.000
>>> >  02-JUL-1960 /  7:  6.000
>>> >  01-AUG-1960 /  8:  7.000
>>> >  31-AUG-1960 /  9:  8.000
>>> >  01-OCT-1960 / 10:  9.000
>>> >
>>> >
>>> > This is expected output :
>>> >
>>> > Here I expect time index to be created like below, at first of every
>>> > month, or else 15th of every month
>>> >
>>> >  01-JAN-1960 /  1:  0.000
>>> >  01-FEB-1960 /  2:  1.000
>>> >  01-MAR-1960 /  3:  2.000
>>> >  01-APR-1960 /  4:  3.000
>>> >  01-MAY-1960 /  5:  4.000
>>> >  01-JUN-1960 /  6:  5.000
>>> >  01-JUL-1960 /  7:  6.000
>>> >  01-AUG-1960 /  8:  7.000
>>> >  01-SEP-1960 /  9:  8.000
>>> >  01-OCT-1960 / 10:  9.000
>>> >
>>> >
>>> > Hope someone can guide me
>>> >
>>> > Thanks in advance.
>>> >
>>> > Have a nice day !
>>> >
>>> > Regards,
>>> > Akshay Hegde
>>> >
>>> >
>>> > Ticket Details
>>> > ===================
>>> > Ticket ID: UXM-196070
>>> > Department: Support netCDF Java
>>> > Priority: Normal
>>> > Status: Open
>>> > Link:
>>> >
>>> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=26875
>>> >
>>> >
>>>
>>>
>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: UXM-196070
>>> Department: Support netCDF Java
>>> Priority: Normal
>>> Status: Open
>>>
>>>
>>>
>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: UXM-196070
>>> Department: Support netCDF Java
>>> Priority: Normal
>>> Status: Open
>>> Link:
>>> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=26875
>>
>>
>>
>



Ticket Details
===================
Ticket ID: UXM-196070
Department: Support netCDF Java
Priority: Normal
Status: Open

Attachment: akshay.ncml
Description: Binary data