> that if one of those underlying files is missing,
> then I assume the whole dataset is unreadable.
I have been wondering that myself. Is there anyone from HDF that can answer
that?
-Roy
> On Jul 13, 2016, at 2:33 PM, dmh@xxxxxxxx wrote:
>
> If I understand it correctly, you are correct: from the API
> point of view, it is invisible. But that is not the issue for me.
> The issue is that now one needs multiple files in order to acccess
> data rather than having everything kept in a single file. That means,
> for example, that if one of those underlying files is missing,
> then I assume the whole dataset is unreadable.
> =Dennis Heimbigner
> Unidata
>
>
> On 7/13/2016 3:25 PM, Roy Mendelssohn - NOAA Federal wrote:
>> Hi Dennis:
>>
>> I asked about this also awhile back. The key question to my mind is how the
>> virtual dataset works in HDF5. Perhaps the interface is such is that it is
>> transparent to higher level calls, much like when you create virtual
>> datasets in TDS. In that case, the calls would be to as if there was one
>> file.
>>
>> -Roy
>>> On Jul 13, 2016, at 2:09 PM, dmh@xxxxxxxx wrote:
>>>
>>> To this point in time, it has been an implicit assumption
>>> that a netcdf dataset encompasses a single file. This functionality
>>> effectively challenges that assumption. So, I would solicit
>>> community input on this. What problems are created if a netcdf
>>> dataset now requires multiple files?
>>>
>>> =Dennis Heimmbigner
>>> Unidata
>>>
>>>
>>> On 7/13/2016 5:09 AM, Eugen Betke wrote:
>>>> Dear NetCDF-Group,
>>>>
>>>> we have been working on NetCDF external link functionality. This allows
>>>> NetCDF applications to create dimension variables which values are
>>>> stored in an external file. Therefore, it uses the HDF5 virtual dataset
>>>> (VDS) functionality. This is useful for, e.g., climate applications that
>>>> rely on a variable per file and timestep configuration. The idea is to
>>>> store the grid in a separate file and link our data to this grid. We
>>>> already have our first working version. You find the patch and the
>>>> examples on our page:
>>>>
>>>> http://wr.informatik.uni-hamburg.de/research/projects/bullio/netcdf_external_links/start
>>>>
>>>>
>>>> Under the hood it uses HDF5 virtual datasets. VDS has the advantage of
>>>> being compatible to the functions that are supported by oridinary
>>>> datasets. Therefore, files containing VDS should be supported by the
>>>> most software.
>>>>
>>>> There is a minor issue related to HDF5, the call H5F_try_close function
>>>> fails, when ncdump trys to read data from an external dimension. So far
>>>> we found a workaround, but we will fix this issue.
>>>>
>>>> It would be great if external link functionality could be supported by
>>>> netCDF at some timepoint. We would like to improve our patch and for
>>>> that reason we need your feedback. If you have some idea to the issue
>>>> above, we would be grateful for each hint.
>>>>
>>>> Regards,
>>>> Eugen
>>>>
>>>> _______________________________________________
>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>> available through the web. Users who post to any of the lists we
>>>> maintain are reminded to remove any personal information that they
>>>> do not want to be made public.
>>>>
>>>>
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S.
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new address and phone***
>> 110 Shaffer Road
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
