Re: [netcdfgroup] Sort of Inverse of netcdf diskless mode

Hi Dennis,

> On Dec 4, 2014, at 12:48 PM, Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> 
> Is this the H5Pset_fapl_core procedure?

        No, I was thinking of this feature set: 
http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf

                Quincey


> =Dennis
> 
> Quincey Koziol wrote:
>> Hi all,
>>      The HDF5 library has a "file image" feature that allows a memory buffer 
>> to be opened as a "file".  The netCDF-4 library could be extended to support 
>> this feature in HDF5.
>> 
>> --
>> Quincey Koziol, The HDF Group
>> Full Contact Info: http://bit.ly/QuinceyKoziol
>> 
>>> Hi Roy:  It's not too big a deal, the netCDF 3 format is nicely 
>>> orthogonal and easy to parse.  The attached perl library is only around 
>>> 350 lines long with adequate comments, but it gives you most basic 
>>> functionality you need for reading.
>>> 
>>> Regards,
>>> 
>>>  Doug
>>> 
>>> On 11/20/14 10:36, Roy Mendelssohn - NOAA Federal wrote:
>>>> Hi Doug:
>>>> 
>>>> Thanks for the comments. I would be interested on seeing the perl code, 
>>>> just to see how complicated it is to do this (though I don?t really know 
>>>> perl).  Fro my purposes. since this is something we will be distributing, 
>>>> we want to keep  what we do to code that is ?native? to the program, in 
>>>> this case R.
>>>> 
>>>> -Roy
>>>> 
>>>> On Nov 20, 2014, at 8:48 AM, Doug Hunt <dhunt@xxxxxxxx> wrote:
>>>> 
>>>>> Hi Roy:  I had a similar problem earlier this year.  I wanted to open a 
>>>>> netCDF 3 file and read from it using the netCDF library when the file was 
>>>>> presented to me in memory instead of as a disk file.
>>>>> 
>>>>> I spent some time examining the netCDF library source and talking to 
>>>>> UNIDATA folks about this and determined that the library was too hard to 
>>>>> change.  The interface depended upon passing in a file name and it would 
>>>>> be too hard to modify it to add an 'ncopen' that took a pointer to memory 
>>>>> instead .
>>>>> 
>>>>> What I ended up doing was writing a reader from scratch in perl (since 
>>>>> this application was in perl using the PDL::NetCDF interface) that reads 
>>>>> the raw netCDF3 binary format.
>>>>> 
>>>>> This has proved to be a good lightweight solution for us, but it does 
>>>>> restrict us to netCDF 3.
>>>>> 
>>>>> I'd be happy to share my perl netCDF 3 reader library if you are 
>>>>> interested.
>>>>> 
>>>>> Incidentally, as a result of this experience, I came away with an 
>>>>> appreciation for the elegance and good design of the netCDF 3 binary 
>>>>> format and a suspicion of the netCDF 4 format as being too complex and 
>>>>> having too much library overhead.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Doug
>>>>> 
>>>>> On 11/19/14 18:41, Roy Mendelssohn - NOAA Federal wrote:
>>>>>> Hi All:
>>>>>> 
>>>>>> Netcdf now has the option of  creating or reading a file entirely in 
>>>>>> memory, a nice option to have. I am wondering if there is any way to do 
>>>>>> sort of the inverse.  Some web libraries allow for a pure binary 
>>>>>> download of a file into memory, so what I have sitting there is 
>>>>>> essentially a binary image of the netcdf file.  Is there any way to open 
>>>>>> that image without writing to disk?
>>>>>> 
>>>>>> As an example, the R httr library allows this.  I can for example 
>>>>>> download to memory a netcdf file from ERDDAP, and if I then do a binary 
>>>>>> save in httr and then read it back in using the ncdf4 package, it all 
>>>>>> works great.   But it would be even better if I could skip the extra 
>>>>>> steps, and ?open? the image in ncdf4 directly  (ncdf4 btw can do 
>>>>>> whatever the netcdf libraries can do, so it is a question of the base 
>>>>>> libraries).
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> -Roy
>> 
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/ 
>> 
>> 



  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: