Hi Ken
As Sourish said, the netCDF self-describing format, which includes the
metadata, is preferred to CSV.
There are many ways to do the conversion: writing a program in C or Fortran
using the netCDF API functions; or with Python, or with Matlab, or IDL.
In my opinion the easiest way is to use NCL.
NCL is a language with data structures and functions that can expedite
the conversion from CSV to netCDF files, if you're willing to install it
and learn the basics.
This will probably be more effective than writing a specific C or Fortran
program to do the conversion using the netCDF API directly
(which will require downloading, installing, and learning anyway).
The bonus is that NCL will allow you to do much more than just
convert CSV to netCDF, including data analysis and data visualization.
Downloading and installing:
https://www.ncl.ucar.edu/Download/
Documentation:
https://www.ncl.ucar.edu/Document/
Here are examples of reading CSV files:
https://www.ncl.ucar.edu/Applications/read_csv.shtml
And examples of writing netCDF files:
https://www.ncl.ucar.edu/Applications/write_netcdf.shtml
I hope this helps,
Gus Correa
On Mon, Mar 18, 2019 at 3:29 PM Sourish Basu <Sourish.Basu@xxxxxxxxxxxx>
wrote:
> Ken,
>
> I belong to a community that likes netcdf files :-) The data you
> describe is perfectly suited for packaging into netcdf. Not knowing the
> specifics of your data, I would say you could have one file per year,
> with variables 'x', 'y', 'lat', 'lon', etc., all of the same length
> (within one file/year, that is). In addition, if you make the common
> dimension of those arrays unlimited, that would allow easy operations by
> command line tools such as netcdf operators.
>
> In general, I find the set of people who like netcdf and the set of
> people who like text files to be completely disjoint. So if you know
> your target is the first set, I'd steer clear of CSV or other text-based
> formats :-)
>
> If you could tell me something more specific about your data, or perhaps
> pass me a sample dataset, I could help you with the packaging, if you'd
> like.
>
> Cheers,
>
> Sourish
>
> On 3/18/19 12:58 PM, Ken Mankoff wrote:
> > Hello,
> >
> > I'm trying to release a data set to a community that likes NetCDF. I'm
> looking for info about if this might be an OK file format given the data
> structure, and if so, how to go about generating the files.
> >
> > I have a 36-year daily time-series (~14,000 times). Presumably I'll
> break this into yearly files to make working with them easier. At each
> time, I have ~20,000 records. Fixed in time is (x,y,lon,lat,elevation), and
> then a varying value. The lon,lat (and associated x,y) are spares points,
> not a grid.
> >
> > Is this a data structure that can be reasonably stored in NetCDF, or
> should I stick to a CSV file, one per year, with 365 columns and
> 20,000+6(header) rows?
> >
> > Thanks,
> >
> > -k.
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
