Re: [netcdfgroup] How to inquire if a NetCDF file is already open for reading or writing?

  • To: Dennis Heimbigner <dmh@xxxxxxxx>, Lucas Villa Real <lucasvr@xxxxxxxxxx>
  • Subject: Re: [netcdfgroup] How to inquire if a NetCDF file is already open for reading or writing?
  • From: Hernan Arango <arango@xxxxxxxxxxxxxxxxxx>
  • Date: Mon, 25 Jan 2021 17:18:44 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marine.rutgers.edu; dmarc=pass action=none header.from=marine.rutgers.edu; dkim=pass header.d=marine.rutgers.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sMWOOdauhYYTAybytewMWDqiDrnIoKbs0702e40VsuY=; b=EfURSKqO4tRIWcoLrtVSCnqCBee8RNXI+RYTjPDeShkH7xSDX0v9EyL41cXPSW9PQjxkK7kJHgyizTpItmMZ6pQXJko997SSo523xbO2ojhcAS3jqINQH/0o1QDNdBa58ZBqr8t1koWpYD8nBkbkk1bEMre0UmGIHwgh3MSKvgaxo2azgjGyyn0YZXMoU/81hDpmDKve6iN6BpbBIgJpwh98LsvTeHTUPrH0xOHCBhXBqieUNjbv9iuF5nFg21vabZzw2z6Fip/pH7q1bFRjuLRm1kafswhMyJfWyckt1NCIuPvEozacW915Ck75mkmS+KBMXiLjnGEChLT0iVXoyw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=See+oMgmc/aipvOLjiDlBlA9v7FKkn/Py/RRElYGyCCzmDdeeG5P9gGapResszBGt2xiC4zppFVjHZ9ZRZpf6WBfg851HOGRR+pZHNddoQTaAb5k8KoVJQgHx7agp2t/oTpws3v90tsIYVhZFurwMJ2Y2sM7CwVchU0QhJPI9O/+H24mtIlmzmDzTXs1sCTu6cHwPgRYzrbZa+k5W5LZM2stjV5X2wg4tGZO/eKncFhIwcdykwVfpCrmJ+v311ML082wrlQSmK7KDpwWVLzPcUcDfYPqJeYmapp7m6pXW99DhuPs2/w9VsPw//Qkp9lSsbrJICcGwFXw75Dmyh1pxA==
  • Authentication-results: ucar.edu; dkim=none (message not signed) header.d=none;ucar.edu; dmarc=none action=none header.from=marine.rutgers.edu;
Hi Dennis,

Thank you for the suggestion.  I already have a wrapper around the create/open 
calls that pass a unique I/O structure for each file type.  My system 
manipulates lots of different NetCDF files that I need to track.  It is 
complicated with parallel I/O that may use or not disjointed and dedicated I/O 
processes. 

Cheers, H

On 1/22/21, 11:25 PM, "Dennis Heimbigner" <dmh@xxxxxxxx> wrote:

    It is doable, I think. Basically you need to walk  the internal list of 
    open files
    looking for a name match. This is probably complicated if mpio is being used
    because that list is duplicated on every processor.
    As an alternative, you could implement wrapper around nc_create and nc_open
    that call the regular nc_create/nc_open, but also keep a list of the 
    names of the
    path arguments.

    =Dennis Heimbigner
       Unidata

    On 1/22/2021 8:35 PM, Hernan Arango wrote:
    >
    > Hi Lucas,
    >
    > Thank you for your answer.  However, what I am specifically asking is: 
    >  Given ONLY the NetCDF dataset filename, is there a function that can 
    > be used in a Fortran code (or a C-binding routine) to inquire if the 
    > filename is open (with a logical true or false result).  If true, 
    >  return the one or more identifiers (vector) associated with the 
    > opening of that specific filename.   Notice that It is possible that 
    > such filename has been opened more than once and not closed.
    >
    > I assume that this is a very difficult question because I don’t have 
    > an idea how to code such function.  Maybe the developers of NetCDF can 
    > tell us if it is possible or not.
    >
    > Cheers, Hernan
    >
    > *From: *Lucas Villa Real <lucasvr@xxxxxxxxxx>
    > *Date: *Friday, January 22, 2021 at 10:17 PM
    > *To: *Hernan Arango <arango@xxxxxxxxxxxxxxxxxx>
    > *Cc: *"netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
    > *Subject: *Re: [netcdfgroup] How to inquire if a NetCDF file is 
    > already open for reading or writing?
    >
    >     ----- Original message -----
    >     From: Hernan Arango <arango@xxxxxxxxxxxxxxxxxx>
    >     Sent by: "netcdfgroup" <netcdfgroup-bounces@xxxxxxxxxxxxxxxx>
    >     To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
    >     Cc:
    >     Subject: [EXTERNAL] [netcdfgroup] How to inquire if a NetCDF file
    >     is already open for reading or writing?
    >     Date: Fri, Jan 22, 2021 6:01 PM
    >
    >
    >     Hello,
    >
    >     I am wondering if there is a function to determine if a NetCDF is
    >     already open for reading or writing based on the filename?   I am
    >     trying to avoid open too many files.   I know that there are ways
    >     in Unix to increase the number of open files, but that’s not what
    >     I am asking.   In parallel I/O running on lots of processes, there
    >     is a need to avoid opening too many files.   I am using both the
    >     standard NetCDF library and the PIO library that is the basis of
    >     the SCORPIO library (https://e3sm.org/scorpio-parallel-io-library
    >     
<https://urldefense.proofpoint.com/v2/url?u=https-3A__e3sm.org_scorpio-2Dparallel-2Dio-2Dlibrary&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=yZCLoPFNF2xM1OIVBISlXxZvERMVKNT1d5PSrOJfnc8&m=QnVlDcV5suiBIhpLT3T5oRt9rSpedaNzD5PPQIaVwoY&s=9fcAm4IjIU6e3FbfD7JzZm28BtCM1zMwV3G7jQ-Bc0A&e=>).
    >     Therefore, I have integer file ID (ncid) and file descriptors of
    >     TYPE (file_desc_t).
    >
    >     I am looking for something like:
    >
    >     ncid=is_nc_open(‘my_file.nc’)
    >
    >     or
    >
    >     fileDesc=is_pio_open(iosystem, my_paralle_file.nc’)
    >
    >     Thank you, H
    >
    > Hi Hernan,
    >
    > If you're on Linux, then one option is to scan the symlinks under 
    > /proc/self/fd looking for a target whose name matches the file name 
    > you're looking for.
    >
    > I'm not sure if ncids have a 1:1 correspondence with HDF5 file ids; if 
    > they do, then perhaps you could also look into H5Fget_obj_count(ncid, 
    > H5F_OBJ_FILE) and/or H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_FILE). See 
    > https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-GetObjCount 
    > <https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-GetObjCount> 
    > for more details on that API.
    >
    > Best regards,
    >
    > Lucas
    >
    >
    >
    >
    > _______________________________________________
    > NOTE: All exchanges posted to Unidata maintained email lists are
    > recorded in the Unidata inquiry tracking system and made publicly
    > available through the web.  Users who post to any of the lists we
    > maintain are reminded to remove any personal information that they
    > do not want to be made public.
    >
    >
    > netcdfgroup mailing list
    > netcdfgroup@xxxxxxxxxxxxxxxx
    > For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/


  • 2021 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: