Re: [netcdfgroup] Sort of Inverse of netcdf diskless mode

  • To: Quincey Koziol <koziol@xxxxxxxxxxxx>
  • Subject: Re: [netcdfgroup] Sort of Inverse of netcdf diskless mode
  • From: Dennis Heimbigner <dmh@xxxxxxxx>
  • Date: Thu, 04 Dec 2014 11:48:54 -0700
Is this the H5Pset_fapl_core
procedure?
=Dennis

Quincey Koziol wrote:
> Hi all,
>       The HDF5 library has a "file image" feature that allows a memory buffer 
> to be opened as a "file".  The netCDF-4 library could be extended to support 
> this feature in HDF5.
> 
> --
> Quincey Koziol, The HDF Group
> Full Contact Info: http://bit.ly/QuinceyKoziol
> 
>> Hi Roy:  It's not too big a deal, the netCDF 3 format is nicely 
>> orthogonal and easy to parse.  The attached perl library is only around 
>> 350 lines long with adequate comments, but it gives you most basic 
>> functionality you need for reading.
>>
>> Regards,
>>
>>   Doug
>>
>> On 11/20/14 10:36, Roy Mendelssohn - NOAA Federal wrote:
>>> Hi Doug:
>>>
>>> Thanks for the comments. I would be interested on seeing the perl code, 
>>> just to see how complicated it is to do this (though I don?t really know 
>>> perl).  Fro my purposes. since this is something we will be distributing, 
>>> we want to keep  what we do to code that is ?native? to the program, in 
>>> this case R.
>>>
>>> -Roy
>>>
>>> On Nov 20, 2014, at 8:48 AM, Doug Hunt <dhunt@xxxxxxxx> wrote:
>>>
>>>> Hi Roy:  I had a similar problem earlier this year.  I wanted to open a 
>>>> netCDF 3 file and read from it using the netCDF library when the file was 
>>>> presented to me in memory instead of as a disk file.
>>>>
>>>> I spent some time examining the netCDF library source and talking to 
>>>> UNIDATA folks about this and determined that the library was too hard to 
>>>> change.  The interface depended upon passing in a file name and it would 
>>>> be too hard to modify it to add an 'ncopen' that took a pointer to memory 
>>>> instead .
>>>>
>>>> What I ended up doing was writing a reader from scratch in perl (since 
>>>> this application was in perl using the PDL::NetCDF interface) that reads 
>>>> the raw netCDF3 binary format.
>>>>
>>>> This has proved to be a good lightweight solution for us, but it does 
>>>> restrict us to netCDF 3.
>>>>
>>>> I'd be happy to share my perl netCDF 3 reader library if you are 
>>>> interested.
>>>>
>>>> Incidentally, as a result of this experience, I came away with an 
>>>> appreciation for the elegance and good design of the netCDF 3 binary 
>>>> format and a suspicion of the netCDF 4 format as being too complex and 
>>>> having too much library overhead.
>>>>
>>>> Regards,
>>>>
>>>>  Doug
>>>>
>>>> On 11/19/14 18:41, Roy Mendelssohn - NOAA Federal wrote:
>>>>> Hi All:
>>>>>
>>>>> Netcdf now has the option of  creating or reading a file entirely in 
>>>>> memory, a nice option to have. I am wondering if there is any way to do 
>>>>> sort of the inverse.  Some web libraries allow for a pure binary download 
>>>>> of a file into memory, so what I have sitting there is essentially a 
>>>>> binary image of the netcdf file.  Is there any way to open that image 
>>>>> without writing to disk?
>>>>>
>>>>> As an example, the R httr library allows this.  I can for example 
>>>>> download to memory a netcdf file from ERDDAP, and if I then do a binary 
>>>>> save in httr and then read it back in using the ncdf4 package, it all 
>>>>> works great.   But it would be even better if I could skip the extra 
>>>>> steps, and ?open? the image in ncdf4 directly  (ncdf4 btw can do whatever 
>>>>> the netcdf libraries can do, so it is a question of the base libraries).
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Roy
> 
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 
> 
> 



  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: