I have tried it again with netcdf-4.1.2-beta1; same results.
I also tried calling nc_set_chunk_cache before nc_open, with args
size=409600 (and also 4096000), nelems=51203, and preemption=0.75.
That didn't seem to change anything for nc_open. Always 20 seconds for
a file the OS hasn't touched in a while.
--Jennifer
p.s. from config.log:
configure:4219: checking whether a default chunk size in bytes was
specified
configure:4229: result: 4194304
configure:4238: checking whether a maximum per-variable cache size for
HDF5 was specified
configure:4248: result: 67108864
configure:4257: checking whether a number of chunks for the default
per-variable cache was specified
configure:4267: result: 10
configure:4276: checking whether a default file cache size for HDF5
was specified
configure:4286: result: 4194304
configure:4295: checking whether a default file cache maximum number
of elements for HDF5 was specified
configure:4305: result: 1009
configure:4314: checking whether a default cache preemption for HDF5
was specified
configure:4324: result: 0.75
On Sep 9, 2010, at 8:06 PM, Denis Nadeau wrote:
Hi Jennifer,
Did you call nc_set_chunk_cache? If not, do you know what the
default cache size was set up to? (Look in your config.log created
after you called configure)
I must admit that 20 seconds is quite long.
Denis
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx
] On Behalf Of Jennifer Adams
Sent: Thursday, September 09, 2010 7:34 PM
To: netCDF Mail List
Subject: [netcdfgroup] nc_open takes a long time to open a big file
Dear Experts,
I'm using netcdf-4.1.1-rc1 and hdf5-1.8.4-patch1 on a 64-bit linux
server running CentOS-5.5.
I have a netcdf-4 file that is 18404502496 bytes large.
My file's dimensions look like this:
lon = 320 ;
lat = 160 ;
lev = 11 ;
time = 1581 ;
It has 7 variables that look like this:
float temp(time, lev, lat, lon) ;
temp:_Storage = "chunked" ;
temp:_ChunkSizes = 1, 1, 160, 320 ;
temp:_DeflateLevel = 1 ;
temp:_Shuffle = "true" ;
and 1 variable that looks like this:
float sfp(time, lat, lon) ;
sfp:_Storage = "chunked" ;
sfp:_ChunkSizes = 1, 160, 320 ;
sfp:_DeflateLevel = 1 ;
sfp:_Shuffle = "true" ;
Is it normal for nc_open to take 20 seconds to open this file before
returning control to my C program?
--Jennifer
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
