I knew that something was left unanswered
@Walter
Is there a way to represent compound HDF5 objects in your scheme? I
see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND.
no, only the atomic types.
see spec
http://www.space-research.org/blog/star_json.html
In any event, FYI, I have implemented a similar thing but in a
narrower domain. I needed to map VOTable [1] (an xml format with a
defined data model for astronomy) to HDF5.
The code is on github [2]. It is mainly meant to be used as a
library, but the code also builds a simple program that converts files
between formats.
ok, thanks,
I did a clone of
https://github.com/Caltech-IPAC/tablator
but it seems there is not a build system?
by the way HDF5 has its own table API, it seems you did not use it
https://support.hdfgroup.org/HDF5/doc/HL/RM_H5TB.html
-Pedro
----- Original Message -----
From: "Walter Landry" <wlandry@xxxxxxxxxxx>
To: <hdf-forum@xxxxxxxxxxxxxxxxxx>; <pedro.vicente@xxxxxxxxxxxxxxxxxx>
Cc: <netcdfgroup@xxxxxxxxxxxxxxxx>; <zender@xxxxxxx>
Sent: Friday, October 14, 2016 2:10 AM
Subject: Re: [Hdf-forum] [netcdfgroup] How to dump netCDF to JSON?
Hi Pedro,
Is there a way to represent compound HDF5 objects in your scheme? I
see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND.
In any event, FYI, I have implemented a similar thing but in a
narrower domain. I needed to map VOTable [1] (an xml format with a
defined data model for astronomy) to HDF5. Along the way, I also
mapped it (with varying levels of fidelity) to JSON, CSV, HTML, FITS,
and plain old ascii. This makes my work very table focused, with a
particular eye towards what relational databases like to output.
The code is on github [2]. It is mainly meant to be used as a
library, but the code also builds a simple program that converts files
between formats. In case you are interested, I am attaching the same
file in plain text, JSON, and HDF5.
Cheers,
Walter Landry
[1] http://www.ivoa.net/documents/VOTable/
[2] https://github.com/Caltech-IPAC/tablator
Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:
Hi Charlie !
So, I am doing that exact same thing.
I wrote
1) The specification to convert netCDF/HDF5 to "a" JSON format (note
the "a" here)
2) I wrote the parsing of the JSON format using an open source C JSON
library (janson).
http://www.digip.org/jansson/
3) left to do: the actual C code of reading/writing netCDF/HDF5 to
JSON and vice-versa (the straightforward part).
The "a" above means that JSON is not really a format in the sense of
netCDF but really a format that allows to define formats , for a lack
of a better
explanation.
This means that anyone that writes this tool has to write code that
write in a particular JSON representation , only valid for that tool.
Like you , I searched and there was not a good one, so I wrote one.
The first criteria was that it had to be obvious for anyone looking at
the JSON text file, that that was indeed a netCDF/HDF5 file: hierarchy
clearly show, metadata and data
clearly shown
My first look was HDF5-JSON
http://hdf5-json.readthedocs.io/en/latest/
but the format seemed like a mess to look at
example
http://hdf5-json.readthedocs.io/en/latest/examples/nullspace_dset.html
and the reader is written in Python
@John Readey
(why Phyton? HDF5 developer tools should be all about writing in
C/C++)
The specification is here
http://www.space-research.org/
Click on menu
"Code blog",
then
"netCDF/HDF5 to JSON and vice-versa"
In the process I learned all about JSON and it is a neat format to
represent data .
In particular, it allows nested structures and arrays, which suits
perfectly for netCDF
here are two nested groups
{
"group_name1":
{
"group_name2": "group"
}
}
a dataset
{
"dset1" : ["dataset", "STAR_INT32", 2, [3, 4], [1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12]]
}
This is still under development,
I would like to make this some kind of "official" netCDF/HDF5 JSON
format for the community, so I encourage anyone to read the
specification
direct link
http://www.space-research.org/blog/star_json.html
If you see any flaw in the design or antything in the design that you
would like to have change please let me know now
At the moment it only (intentionally) uses common generic features of
both netCDF and HDF5, which are the numeric atomic types and strings.
Enjoy
----------------------
Pedro Vicente
pedro.vicente@xxxxxxxxxxxxxxxxxx
http://www.space-research.org/
----- Original Message -----
From: "Charlie Zender" <zender@xxxxxxx>
To: "netCDF Mail List" <netcdfgroup@xxxxxxxxxxxxxxxx>
Sent: Thursday, October 13, 2016 11:10 PM
Subject: [netcdfgroup] How to dump netCDF to JSON?
Hello netCDFers,
A project I am working on wants to convert netCDF files to JSON.
The requirements are to dump an arbitrary netCDF-extended file
(with groups but without funky vlen/compound types) to JSON.
The first few solutions that we googled (ncdump-json, netcdf2json.py)
do not satisfy these requirements. What is the most robust and easy
command-line tool (not web-service) that dumps netCDF to JSON?
Ideally it would be somewhat configurable like ncdump -h/-x or
ncks --cdl/--xml.
Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@xxxxxxxxxxxxxxxxxx
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--------------------------------------------------------------------------------
| object | ra | dec | htm20
| htm7 | htm3 | shtm20 | shtm7 | shtm3 | flags |
SSO |
| char | double | real | ulong
| uint | ushort | long | int | short | byte |
bool |
118289arstratraetratratsrastratsrastrats 359.88703 50.832570
16446744073709551616 3294967296 12000 8223372036854775808
1147483648 12000 122 0
113368 344.41273 -29.622250
8446744073709551616 294967296
43002 -7223372036854775808 -2047483648 13002 0xf2 true
113368 344.41273 -29.622250
8446744073709551616 294967296
43002 -7223372036854775808 -2047483648 -23002 211 False
113368 344.41273 -29.622250
8446744073709551616 294967296
43002 -7223372036854775808 -2047483648 -31002 211 1
--------------------------------------------------------------------------------
{
"VOTABLE":
{
"<xmlattr>":
{
"version": "1.3",
"xmlns:xsi": "http:\/\/www.w3.org\/2001\/XMLSchema-instance",
"xmlns": "http:\/\/www.ivoa.net\/xml\/VOTable\/v1.3",
"xmlns:stc": "http:\/\/www.ivoa.net\/xml\/STC\/v1.30"
},
"RESOURCE":
{
"TABLE":
{
"FIELD":
{
"<xmlattr>":
{
"name": "object",
"datatype": "char",
"arraysize": "*"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "ra",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "dec",
"datatype": "double"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm20",
"datatype": "ulong"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm7",
"datatype": "uint"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "htm3",
"datatype": "ushort"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm20",
"datatype": "long"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm7",
"datatype": "int"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "shtm3",
"datatype": "short"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "flags",
"datatype": "unsignedByte"
}
},
"FIELD":
{
"<xmlattr>":
{
"name": "SSO",
"datatype": "boolean"
}
},
"DATA":
{
"TABLEDATA":
[
[
"118289arstratraetratratsrastratsrastrats",
"359.88702999999998",
"50.832569999999997",
"16446744073709551616",
"3294967296",
"12000",
"8223372036854775808",
"1147483648",
"12000",
"0x7a",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"13002",
"0xf2",
"1"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-23002",
"0xd3",
"0"
],
[
"113368",
"344.41273000000001",
"-29.622250000000001",
"8446744073709551616",
"294967296",
"43002",
"-7223372036854775808",
"-2047483648",
"-31002",
"0xd3",
"1"
]
]
}
}
}
}
}
--------------------------------------------------------------------------------
?HDF
ÿÿÿÿÿÿÿÿ?0ÓGâAOHDR TrXTrXTrXTrXxÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
tableÃD"%=?OHDR!TrXTrXTrXTrX 6
Xnull_bitfield_flags:object(ra* ?@4 4ÿdec2 ?@4
4ÿhtm20:@htm7B htm3Fshtm20H@shtm7P
shtm3TflagsVSSOW
0`ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
¼ -METADATA960namevalueattributes 96
namevalue4)
±118289arstratraetratratsrastratsrastrats"?^F1~v@»Ð\§'jI@8±~'>ä6eÄà.oXLI r6eDà.z113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.Ê2ò113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.&¦Ó113368@ÁÅSs?u@zï§ÆKY=ÀvûÜ8uØ"ú§ÈNgmÁ>áõ.æ?Ó
