Re: [netcdfgroup] Need help to convert .csv files to netCDF files

Dumindu,


You can paste a file name together in a few loops like so (check for mistakes!):


yb=1998; ye=2000; % for the years yb through to ye
mb=1; me=12;   %% month begin, month end

%% day numbers in the year for times of data
md=[31;28;31;30;31;30;31;31;30;31;30;31];   % days per month
yd=1; for i=2:12; yd(i)=yd(i-1)+md(i-1); end    % doy of month start
mc=['jan';'feb';'mar';'apr';'may';'jun';'jul';'aug';'sep';'oct';'nov';'dec']

for iy=yb:ye % for the years
cy=sprintf('%4d',iy)   %formated print to the string

for im=mb:me  % for the months
  if(mod(iy,4)==0 && im==2 ) mtyd=mtyd+1; end   % account for leap years

  cm1=sprintf('%3d',100+im);
  cm=cm1(2:3);

  for id=1:md(im)  %
    for ic=1:8  % eight files per day?
    cd1=sprintf('%3d',10000+id*100+ic*3);
    cd=cd1(2:5);
    fname=[‘TRMM_’,cy(3:4),’_’,cm,’_’,cd,'_newntcl.csv']
% Open the file and assign it a file ID
    fid = fopen(fname,'r');

…
End all loops


If that does not work, then pipe all fine names into a file using ls and read 
name by name of that file in a loop over all file names. This has the advantage 
that you will not try to read missing files and it does not matter whether 
there is a pattern to the file name or not.

Best wishes,
Joerg



From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx 
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Dumindu Jayasekera
Sent: 12 March 2015 01:11
To: Chris Barker
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Need help to convert .csv files to netCDF files

Chris,

Each time step is a different file. Original final name of the "TEST_file.csv 
is TRMM_1998_01_0100_newntcl.csv. So the next file for next time step is 
TRMM_1998_01_0103_newntcl.csv, TRMM_1998_01_0106_newntcl.csv, 
TRMM_1998_01_0109_newntcl.csv  and so on. I have the time steps until 
TRMM_1998_02_1512_newntcl.csv.

Do you know how to loop through file names and store in arrays?

Thanks again.




On Wed, Mar 11, 2015 at 3:19 PM, Chris Barker 
<chris.barker@xxxxxxxx<mailto:chris.barker@xxxxxxxx>> wrote:
Sorry, I really dont have time to do it for you, but a couple hints:

IF you're lucky, you can "reshape" the input arrays in one step:

data = np.loadtxt('TEST_file.csv', delimiter=',')

lat = data[:,0] # the first column -- if that is latitude

lat = lat.reshape( (num_times, num_lats, num_lons) )

That _might put it all in the right order, depending on hoe it's written to the 
file. If that doesn't work, this might:

lat = lat.reshape( (num_times, num_lats, num_lons),  order='F')

If that doesn't work,  then you may have to loop through all of by hand, in the 
right order:

new_data = np.zeros( (num_times, num_lats, num_lons) )
i = 0
for t in range(num_times):
    for lat in range(num_lats):
        for lon in range(num_lons):
            new_data[t, lat, lon] = old_data[i]
            i += 1

You may need to change the order of those loops to match your data.

Also,m I see three columns, not four in your sample file -- is each time step 
in a different file? That wold require you to loop through all the files to 
load each time...

Take a look at python and numpy tutorials online to learn a bit more about all 
this.

-Chris




On Wed, Mar 11, 2015 at 12:03 PM, Dumindu Jayasekera 
<d.jayasekera@xxxxxxxxxxxxxxxxx<mailto:d.jayasekera@xxxxxxxxxxxxxxxxx>> wrote:
Chris,

Thanks again. I am unable to create the 3D array since I have limited knowledge 
in python.

I have attached the csv files, python scripts and sample netCDF file.

Any help is appreciated.





On Wed, Mar 11, 2015 at 8:45 AM, Chris Barker 
<chris.barker@xxxxxxxx<mailto:chris.barker@xxxxxxxx>> wrote:
On Tue, Mar 10, 2015 at 5:50 PM, Dumindu Jayasekera 
<d.jayasekera@xxxxxxxxxxxxxxxxx<mailto:d.jayasekera@xxxxxxxxxxxxxxxxx>> wrote:
Thanks again Chris.

I think I dont need to core metadata. I was able to produce the .nc file using 
the code below (as you suggested).


As Rich suggests, you may   be better off using onf the libraries suggested, 
Iris, in particular will get all the complex CF metadaat stuff right for you.

But you're this close, so...

But, how can I modify to rename the var1 = lat, var2 = lon, var3 = 
precipitation in the code below.?

 From the nc file you sent, it looks like you want the precipitiaon to be a 3-d 
array: (time X ltitude X longitude), so:


# this load the file into a Nx3 array (three columns)
data = np.loadtxt('TEST_file.csv', delimiter=',')

note -- you dont have a 3-d array here, you have a 2-d array, which is really 
three vectors -- you will need to re-shuffle these to get the 3-d array you 
want, and the time, lat, and long vectors.... but assumign you've done that:


you'll need three dimensions:
    time = ds.createDimension('time', num_times)
    lat = ds.createDimension('latitude', num_latitude)
    lon = ds.createDimension('longitude', num_longitude)

Then you want your 3-d precip variable -- I"d call it something more readable 
than "r", but maybe you need that...

r = ds.createVariable('r', np.float32, ('time', 'latitude', 'longitude'))
var[:] = data[:,:,:]
## adds some attributes
var.units = 'mm'
var.long_name = 'Precipitation'
... and the others that you need.

HTH,

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959<tel:%28206%29%20526-6959>   voice
7600 Sand Point Way NE   (206) 526-6329<tel:%28206%29%20526-6329>   fax
Seattle, WA  98115       (206) 526-6317<tel:%28206%29%20526-6317>   main 
reception

Chris.Barker@xxxxxxxx<mailto:Chris.Barker@xxxxxxxx>




--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959<tel:%28206%29%20526-6959>   voice
7600 Sand Point Way NE   (206) 526-6329<tel:%28206%29%20526-6329>   fax
Seattle, WA  98115       (206) 526-6317<tel:%28206%29%20526-6317>   main 
reception

Chris.Barker@xxxxxxxx<mailto:Chris.Barker@xxxxxxxx>

  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: