Hi Don,
Do your files happen to have an unlimited dimension when it is not required?
In the past we've had a performance issues dealing with static data sets only
to later realize the slow load times where due to the reading of data
associated with an unlimited dimension (i.e. "time"). When a dimension is
unlimited the values associated with it are stored sparsely though out the the
file. Converting the unlimited dimension to fixed significantly increased
time-to-open these files (i.e. the values for the "time" axis are stored
contiguously, no longer sparsely). You'll want unlimited if you intend to
append data along that dimension to the file in the future, otherwise make sure
it's fixed if you are concerned about performance on initial open.
Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI 53562
On Jun 21, 2012, at 4:13 PM, Don Murray wrote:
> Just as a followup, the attached program tests the speed of opening a file
> using the method in FeatureScan vs. GridDataset.open. In my test, the latter
> is actually faster by a few milliseconds. The real slowdown is the initial
> os caching of the file (in this case a 3.3 GB file). Once the file is in the
> OS cache, both methods are pretty quick.
>
> Thanks to John (and Roland) for their help.
>
> Don
>
> On 6/20/12 8:14 PM, John Caron wrote:
>> On 6/19/2012 3:19 PM, Don Murray wrote:
>>> Hi-
>>>
>>> I have a bunch of netCDF files and I want to quickly determine whether
>>> they are grids, trajectories, or point features. For grids, I've been
>>> using GridDataset gds = GridDataset.open(path) and catch the exception
>>> if it's not a grid, but for a 3.3 GB file, that can take 2 minutes (or
>>> longer) to open and create the dataset if it is a grid. I was
>>> wondering if there's a quicker method of determining the feature type
>>> of a netCDF file.
>>>
>>> Thanks for your help.
>>>
>>> Don
>>
>> Hi Don:
>>
>> The most convenient thing is to use ToolsUI / FeatureTypes /
>> FeatureScan, and give it a file or directory. It will try to figure out
>> the type and report on what it finds.
>>
>> The code is in ucar.nc2.ft.scan.FeatureScan.java, you can copy the parts
>> you need.
>>
>> Its an ongoing process, i think im not doing it as well as it can be
>> done. Send me reports on files it misidentifies.
>>
>> John
>>
>> _______________________________________________
>> netcdf-java mailing list
>> netcdf-java@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
>
>
> <TestOpen.java>_______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
