Re: [netcdf-java] How to create a structure in netcdf4

  • To: José Évora <jose.evora@xxxxxxxx>
  • Subject: Re: [netcdf-java] How to create a structure in netcdf4
  • From: Christian Ward-Garrison <cwardgar@xxxxxxxx>
  • Date: Tue, 8 Dec 2015 16:57:17 -0700
Hi José,

Once again, sorry for the delay. I've modified my last example to
incorporate the changes you requested [1]. A few things to note:

1. The "Other" dimension is now shared instead of private.
NetcdfFileWriter.addDimension creates a dimension and adds it to a Group.
When Dimensions are contained by Groups, they should *always* be shared. In
fact, NetcdfFileWriter.addDimension shouldn't even have the isShared
parameter, as it is confusing and error-prone. In v5.0.0, it won't [2].
Thanks for bringing this to my attention!

2. In the first example, the Date1 and Date2 variables use *anonymous
dimensions*. Anonymous dimensions are both private (i.e. contained by a
Variable, not a Group) and nameless. NetcdfFileWriter does not provide a
direct way to create them, so you'll have to do a little more work, as I've
shown.

3. In the first example, it looks like the "CHANGELOG=2" snippet may
indicate a private, *named* dimension. If that's the case, create it the
same way you did the anonymous dimensions but just – you know – give it a
name.

Cheers,
Christian

[1] https://gist.github.com/cwardgar/35427c2e245ec0157416
[2] https://github.com/Unidata/thredds/pull/326

On Tue, Nov 24, 2015 at 1:23 AM, José Évora <jose.evora@xxxxxxxx> wrote:

> Hi Christian,
>
> Thank you very much, it worked perfectly!. However, I'm trying to
> replicate the output as expected for a netcdf reader of a specific kind of
> data. When I read a netcdf that is accepted by the specific reader using
> the netcdf java library, this is what I obtain (I've have adapted the
> content to our toy case, but it can be extrapolated to the real case):
>
> netcdf res/in original.nc {
>>   dimensions:
>
>     City = 1;
>
>     Other = UNLIMITED;   // (2 currently)
>>     ...
>>   variables:
>
>     String City(City=1);
>
>
>
>     Structure {
>>       char Date1(30);
>>       char Date2(53);
>>     } CHANGELOG(CHANGELOG=2);
>>       :_ChunkSizes = 1; // int
>> }
>
>
> With the code you provided me, I obtain:
>
>  netcdf res/mockSimulation2.nc {
>>   dimensions:
>>     City = 1;
>>     Other-Entry_Dim0 = 1;
>>     Other = 1;
>>     strlen = 30;
>>   variables:
>>     String City(City=1);
>>     Structure {
>>       char Date1(30);
>>       char Date2(30);
>>     } Other-Entry(Other-Entry_Dim0=1);
>>
>> }
>
>
> I only did one change to what you sent me at line 26:
>
> Structure otherVar = (Structure) ncFile.addVariable(root, "Other-Entry",
>> STRUCTURE, singletonList(other));
>
>
> I don't know how I can:
>
> 1) Avoid having a dimension strlen
> 2) Avoid having Other-Entry_Dim0
>
> Thanks in advance!
>
> José Évora Gómez
> Instituto Universitario SIANI
> Universidad de Las Palmas de GC
> @j_evora
>
> 2015-11-23 6:59 GMT+00:00 Christian Ward-Garrison <cwardgar@xxxxxxxx>:
>
>> Hi José,
>>
>> Sorry for the late response—hopefully this is still useful to you.
>>
>> Writing structures to NetCDF-4 is a recent addition to our library and
>> hasn't been tested as comprehensively as we'd like, especially since most
>> of our users that are writing NetCDF-4 are only utilizing the classic data
>> model (i.e. no structures). Your example uncovered multiple bugs [1][2][3],
>> so thank you for that!
>>
>> That said, it is still possible to make your example work, although the
>> admittedly limited documentation makes it tricky to get right. I've
>> corrected your NetCdfCreatorLite2 class and posted it here [4]. Notice that
>> the "Other" dimension is no longer unlimited and that I'm recalculating the
>> structure's size after adding members to it.
>>
>> Also, for future reference, the latest versions of our library artifacts
>> are not available on Maven Central. Instead, grab them from our
>> organization's repository, as described here [5].
>>
>> Cheers,
>> Christian
>>
>> [1] https://github.com/Unidata/thredds/issues/296
>> [2] https://github.com/Unidata/thredds/issues/298
>> [3] https://github.com/Unidata/thredds/issues/299
>> [4] https://gist.github.com/cwardgar/7aa866886468e97c8eea
>> [5]
>> https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/BuildDependencies.html
>>
>> On Thu, Nov 5, 2015 at 8:01 AM, José Évora <jose.evora@xxxxxxxx> wrote:
>>
>>> The dependency is being taken from Maven central. However, I have tried
>>> this jar instead of maven:
>>>
>>> ftp://ftp.unidata.ucar.edu/pub/netcdf-java/v4.6/netcdfAll-4.6.jar
>>>
>>> And I get exactly the same problems in both java files.
>>>
>>> José Évora Gómez
>>> Instituto Universitario SIANI
>>> Universidad de Las Palmas de GC
>>> @j_evora
>>>
>>> 2015-11-05 14:09 GMT+00:00 John Caron <jcaron1129@xxxxxxxxx>:
>>>
>>>> what repo is
>>>>
>>>> <dependency>
>>>> <groupId>edu.ucar</groupId>
>>>> <artifactId>netcdf4</artifactId>
>>>> <version>LATEST</version>
>>>> </dependency>
>>>>
>>>> coming from?
>>>>
>>>>
>>>>
>>>> On Thu, Nov 5, 2015 at 12:13 AM, José Évora <jose.evora@xxxxxxxx>
>>>> wrote:
>>>>
>>>>> When maven resolves the code quoted, this is the library that
>>>>> downloads:
>>>>>
>>>>> Maven: edu.ucar:netcdf4:4.5.5
>>>>>
>>>>>
>>>>> Version seems to be 4.5.5 for the netcdf java library. For the netcdf
>>>>> C libraries, I downloaded and installed: netCDF 4.3.3.1. Specifically, 
>>>>> this
>>>>> file:
>>>>>
>>>>>
>>>>> http://www.unidata.ucar.edu/downloads/netcdf/ftp/netCDF4.3.3.1-NC4-DAP-64.exe
>>>>>
>>>>> Thanks,
>>>>>
>>>>> José Évora Gómez
>>>>> Instituto Universitario SIANI
>>>>> Universidad de Las Palmas de GC
>>>>> @j_evora
>>>>>
>>>>> 2015-11-04 23:33 GMT+00:00 John Caron <jcaron1129@xxxxxxxxx>:
>>>>>
>>>>>> Hi Jose, what version of netcdf-java and netcdf C libraries are you
>>>>>> using? John
>>>>>>
>>>>>> On Wed, Nov 4, 2015 at 12:54 AM, José Évora <jose.evora@xxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'm trying to create a structure (also know compounded type). In
>>>>>>> this email I attach two different ways I've tried to do it and their
>>>>>>> respective error message. This is the dependency I'm using
>>>>>>>
>>>>>>> <dependency>
>>>>>>>> <groupId>edu.ucar</groupId>
>>>>>>>> <artifactId>netcdf4</artifactId>
>>>>>>>> <version>LATEST</version>
>>>>>>>> </dependency>
>>>>>>>
>>>>>>>
>>>>>>> *1) NetCdfCreatorLite.java*
>>>>>>>
>>>>>>> The execution throws a NullPointerException at
>>>>>>>
>>>>>>> Structure otherVar = ncFile.addStructure(root,
>>>>>>>> root.getParentStructure(), "Other", singletonList(other));
>>>>>>>
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>>>> at ucar.nc2.NetcdfFileWriter.addStructure(NetcdfFileWriter.java:584)
>>>>>>>> at org.mock.NetCdfCreatorLite.main(NetCdfCreatorLite.java:27)
>>>>>>>
>>>>>>>
>>>>>>> *2) NetCdfCreatorLite2.java*
>>>>>>>
>>>>>>> According to the java doc, the method addstructure shouldn't be
>>>>>>> used. Then, in this second version, I uses a addVariable method to 
>>>>>>> create
>>>>>>> the structure:
>>>>>>>
>>>>>>> Structure otherVar = (Structure) ncFile.addVariable(root, "Other",
>>>>>>>> STRUCTURE, singletonList(other));
>>>>>>>
>>>>>>>
>>>>>>> The exeception given this time is:
>>>>>>>
>>>>>>> Exception in thread "main" java.io.IOException: nc_put_vars: -49:
>>>>>>>> NetCDF: Variable not found grpid=65536 objid=1
>>>>>>>> at ucar.nc2.jni.netcdf.Nc4Iosp.writeCompoundData(Nc4Iosp.java:2895)
>>>>>>>> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2839)
>>>>>>>> at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2721)
>>>>>>>> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:934)
>>>>>>>> at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:918)
>>>>>>>> at org.mock.NetCdfCreatorLite2.main(NetCdfCreatorLite2.java:46)
>>>>>>>
>>>>>>>
>>>>>>> At line:
>>>>>>>
>>>>>>> ncFile.write(otherVar, otherStructure);
>>>>>>>
>>>>>>>
>>>>>>> If I comment the creation of structures, the application executes
>>>>>>> perfectly. Even more, by only adding the addVariable of a structure 
>>>>>>> line, I
>>>>>>> get this exception, even though I only write the cityVar at line 50 .
>>>>>>>
>>>>>>> What is wrong? Can you please provide me with an example of use of
>>>>>>> the structures?.
>>>>>>>
>>>>>>> Best regards, thank you very much,
>>>>>>>
>>>>>>> José Évora Gómez
>>>>>>> Instituto Universitario SIANI
>>>>>>> Universidad de Las Palmas de GC
>>>>>>> @j_evora
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netcdf-java mailing list
>>>>>>> netcdf-java@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> netcdf-java mailing list
>>> netcdf-java@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>
  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: