Re: [thredds] size limit for NcML values element entries?

Hi Steve, Rich,

Unfortunately, my StackOverflow reputation is so low (as I've only
lurked on SO till now) that I can't comment on the question. So, I'll
ask a few questions here.

Steve, Is your data a "regular" grid or an "unstructured" grid? I'm
guessing a "regular" grid since you are trying to use a featureCollection.

Though the CF convention requires lat/lon arrays, the netCDF-Java
library does not. If there is grid mapping information, it can do all it
needs with x/y. So, I suspect the missing lat/lon are not the reason
that the featureCollection is failing.

Have you tried to open your datasets in ToolsUI (available on the
netCDF-Java page [1])? Try it in the "CoordSys" and then the
"FeatureTypes - Grids" tabs. In "CoordSys" you should see the coordinate
axes listed, above those the coordinate system(s), and above those the
data variables.

If that doesn't work, you might need to use NcML to add some attributes
and such to make the grid mapping information (or something) CF compliant.

Hope that helps,

Ethan

[1] http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm

On 2/8/2013 11:08 AM, Signell, Richard wrote:
> Steve,
> Let's move this discussion off the thredds mailing list to
> stackoverflow.
> http://stackoverflow.com/questions/14776450/is-there-a-hardwired-limit-to-the-ncml-thredds-values-element/14778665#14778665
> 
> Ask your followup there.
> 
> -Rich
> 
> On Fri, Feb 8, 2013 at 12:29 PM, Steven Baum <baum@xxxxxxxxxxxxxxxx> wrote:
>>
>> Rich,
>>
>> I just posted the question on stackoverflow - a very interesting and useful 
>> site - under
>> the netcdf category since that was the only one that was both applicable and 
>> allowed.
>>
>> As for your suggestion of using a union, I set up a very long script last 
>> night to use
>> the NCOs to append the lon/lat arrays to all of the history files, although 
>> it's taking
>> a good long while to append them to the 13GB history files as well as supply 
>> the
>> required attribute information.  It's only gotten through about 30 of the 
>> files, so if I
>> stop it and try the union thing, what will happen if I attempt to union the 
>> files to
>> which I've already added the lon/lat fields with the NetCDF file containing 
>> the lon/lat
>> files?  That is, what happens when the same field is present within both 
>> files that
>> you're trying to union?
>>
>> Steve
>>
>> On Fri, 8 Feb 2013 07:46:26 -0500, Signell, Richard wrote
>>> Steve,
>>>
>>> I don't know the answer to your question about the size limit for
>>> <values>, but there is another way to solve your problem..
>>>
>>> You can write your lon,lat values to a netcdf file, and then use a
>>> "union" aggregation that aggregates the "joinExisting"
>>> aggregation with your grid file.
>>>
>>> For example, if you have a bunch of files like:
>>> mod_his_0001.nc
>>> mod_his_0002.nc
>>> ...
>>> and you create a netcdf file that contains the lon,lat variables:
>>> lonlat.nc
>>>
>>> you can aggregate them together like so:
>>>
>>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>  <!-- union the (1) time aggregation with the (2) grid file -->
>>>  <aggregation type="union">
>>>   <!-- (1) time aggregate files like "mod_his_0001.nc" -->
>>>   <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>    <aggregation dimName="time" type="joinExisting">
>>>     <scan
>>>      location="/home/baum/models/run01/"
>>>      regExp=".*mod_his_[0-9]{4}\.nc$"/>
>>>    </aggregation>
>>>   </netcdf>
>>>   <!-- (2) grid file -->
>>>   <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";
>>>    location="/home/baum/models/lonlat.nc"/>
>>>  </aggregation>
>>> </netcdf>
>>>
>>> Make sense?
>>>
>>> -Rich
>>>
>>> P.S. Can you please ask the same question on stackoverflow.com and tag
>>> with "netcdf" and "xml"?  I think as a community we should start to
>>> get info like this captured in a place where others can comment and
>>> improve the answers, and best answers can rise to the top.  It's a lot
>>> easier than wading through mailing lists.  If you ask it, I promise to
>>> answer.   And I need to get up to 1500 points so I can add an "ncml"
>>> tag.   ;-)
>>>
>>> On Thu, Feb 7, 2013 at 5:44 PM, Steven Baum <baum@xxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> Is there a limit to the size or number of entries that can be placed 
>>>> within the values
>>>> NcML element?  I'm attempting to get a set of NetCDF files without lon/lat 
>>>> variables -
>>>> but with x/y variables - to be read as a featureCollection by adding NcML 
>>>> entries of the
>>>> form:
>>>>
>>>>       <netcdf 
>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>>
>>>>         <variable name="lat_rho" shape="eta_rho xi_rho" type="float">
>>>>           <attribute name="units" type="String" value="degrees_north" />
>>>>             <values>[671x191 blankspace-separated latitudes here]</values>
>>>>         </variable>
>>>>
>>>> The lat and lon variables exist in a separate file and I've extracted them 
>>>> into single
>>>> strings using ncks which I've stuck between appropriate <values></values> 
>>>> locations.
>>>>
>>>> The error messages in featureCollectionScan.log are:
>>>>
>>>> [2013-02-07T15:15:37.386-0600] ERROR ucar.nc2.ft.fmrc.Fmrc: makeFmrcInv
>>>> java.lang.NullPointerException
>>>>
>>>> ....
>>>>
>>>> [2013-02-07T15:15:37.388-0600] ERROR ucar.nc2.ft.fmrc.Fmrc:
>>>> /raid/data/txla_nesting6/.*\.nc$: makeFmrcInv failed
>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>
>>>> ....
>>>>
>>>> [2013-02-07T15:15:37.694-0600] WARN  ucar.nc2.ft.fmrc.GridDatasetInv: 
>>>> GridDatasetInv
>>>> using gds.getStartDate() for run date =%s
>>>> [2013-02-07T15:15:37.694-0600] ERROR ucar.nc2.ft.fmrc.Fmrc: makeFmrcInv
>>>> java.lang.NullPointerException
>>>>
>>>> ....
>>>> [2013-02-07T15:15:37.695-0600] ERROR ucar.nc2.ft.fmrc.Fmrc:
>>>> /raid/data/txla_nesting6/.*\.nc$: makeFmrcInv failed
>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>> --
>>> Dr. Richard P. Signell   (508) 457-2229
>>> USGS, 384 Woods Hole Rd.
>>> Woods Hole, MA 02543-1598
>>
>>
>> --
>> Open WebMail Project (http://openwebmail.org)
>>
> 
> 
> 

-- 
Ethan Davis                                       UCAR Unidata Program
edavis@xxxxxxxxxxxxxxxx                    http://www.unidata.ucar.edu



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: