Re: [netcdfgroup] nc_open takes a long time to open a big file

Hi Jennifer,
 
I have the same result on handling of a complex file. This is a NetCDF
property: nc_open scans the whole file and builds internal structures
containing all dimensions and variables in the file.
This is the price for luxury not to bother about closing open objects as
in HDF API.
If you are not happy about it use HDF API: it does exactly what you ask
for without spare operations.
 
Regards,
Sergei 
 
 -----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jennifer
Adams
Sent: 10 September 2010 01:42
To: netCDF Mail List
Subject: Re: [netcdfgroup] nc_open takes a long time to open a big file



        I have tried it again with netcdf-4.1.2-beta1; same results.  

        I also tried calling nc_set_chunk_cache before nc_open, with
args size=409600 (and also 4096000), nelems=51203, and preemption=0.75.
That didn't seem to change anything for nc_open. Always 20 seconds for a
file the OS hasn't touched in a while.  

        --Jennifer

        p.s. from config.log:
        configure:4219: checking whether a default chunk size in bytes
was specified
        configure:4229: result: 4194304
        configure:4238: checking whether a maximum per-variable cache
size for HDF5 was specified
        configure:4248: result: 67108864
        configure:4257: checking whether a number of chunks for the
default per-variable cache was specified
        configure:4267: result: 10
        configure:4276: checking whether a default file cache size for
HDF5 was specified
        configure:4286: result: 4194304
        configure:4295: checking whether a default file cache maximum
number of elements for HDF5 was specified
        configure:4305: result: 1009
        configure:4314: checking whether a default cache preemption for
HDF5 was specified
        configure:4324: result: 0.75



        On Sep 9, 2010, at 8:06 PM, Denis Nadeau wrote:


                Hi Jennifer,
                Did you call nc_set_chunk_cache?  If not, do you know
what the default cache size was set up to?  (Look in your config.log
created after you called configure)
                I must admit that 20 seconds is quite long.
                Denis
                From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jennifer
Adams
                Sent: Thursday, September 09, 2010 7:34 PM
                To: netCDF Mail List
                Subject: [netcdfgroup] nc_open takes a long time to open
a big file
                Dear Experts,
                I'm using netcdf-4.1.1-rc1 and hdf5-1.8.4-patch1 on a
64-bit linux server running CentOS-5.5.
                I have a netcdf-4 file that is 18404502496 bytes large. 
                My file's dimensions look like this:
                        lon = 320 ;
                        lat = 160 ;
                        lev = 11 ;
                        time = 1581 ;
                It has 7 variables that look like this:
                   float temp(time, lev, lat, lon) ;
                                temp:_Storage = "chunked" ;
                                temp:_ChunkSizes = 1, 1, 160, 320 ;
                                temp:_DeflateLevel = 1 ;
                                temp:_Shuffle = "true" ;
                and 1 variable that looks like this: 
                    float sfp(time, lat, lon) ;
                                sfp:_Storage = "chunked" ;
                                sfp:_ChunkSizes = 1, 160, 320 ;
                                sfp:_DeflateLevel = 1 ;
                                sfp:_Shuffle = "true" ;
                Is it normal for nc_open to take 20 seconds to open this
file before returning control to my C program? 
                --Jennifer
                --
                Jennifer M. Adams
                IGES/COLA
                4041 Powder Mill Road, Suite 302
                Calverton, MD 20705
                jma@xxxxxxxxxxxxx


        --
        Jennifer M. Adams
        IGES/COLA
        4041 Powder Mill Road, Suite 302
        Calverton, MD 20705
        jma@xxxxxxxxxxxxx





        Click here
<https://www.mailcontrol.com/sr/qZ+83bK30V3TndxI!oX7Umrf+X9KFnd0BK9MqNtI
AlohmNeiQKx8Hz0mc0MEr5X25CtN22G6Q+xhjvXYAQ53ow==>  to report this email
as spam.

  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: