NOTE: The decoders mailing list is no longer active. The list archives are made available for historical reasons.
hiya, i'm a little confused about how a 1 mg file gets expanded to 1 gig. the usual expansion from grib 1 to netcdf is about 10x, grib 2 can be 40x. i could see a larger file if navigation information is added, but not more than 15x. But Glenn did state the resulting file was only about 3-4 times, that sounds reasonable. Could you point me to a file so i can take a look? robb...
---------- Forwarded message ---------- From: Glenn Rutledge <glenn.rutledge@xxxxxxxx> Date: Tue, 19 Oct 2004 09:51:47 -0400 Subject: Re: FW: Re: NCEP North American Reanalysis (fwd) To: goodalljl@xxxxxxxxxxxxxxx Cc: russ@xxxxxxxxxxxxxxxx, ben@xxxxxxxxxxxxxxxx, maidment@xxxxxxxxxxxxxxx Hi Jon, Just to clarify- we were trying to provide "a service" for users that converts these data to NetCDF, then compress it. The mg to gb was an intermediary file that the routines apparently constructs to do the conversion - then it pares the data into NetCDF with is only about 2- 4 times bigger. Not several orders of magnitude. So- we are now just developing an informational page that points to Unidata web sites that users can download the utilities to do this conversion themselves. So- it's just not that bad. Hope this helps. Glenn Jon Goodall wrote: Russ and Ben, We are working with the NCDC to convert NARR reanalysis data from Grib to netCDF. They reported that initial attempts to convert a 1mb NARR file in grib resulted in a 1gb NetCDF file. Dr. Madment wanted me check with you to see if this sounds right. I'm sure it depends on the specifics of the conversion routine, which Glenn Rutledge would know. We want to use netCDF to store the NARR data, but can not justify a factor of 1,000 storage increase. Jon Jon Goodall Center for Research in Water Resources University of Texas at Austin ----Original Message Follows---- From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> To: goodalljl@xxxxxxxxxxxxxxx Subject: Re: NCEP North American Reanalysis (fwd) Date: Tue, 12 Oct 2004 09:46:05 -0400 Hi Jon, We just developed the capability to do this and find that it will not be wise for me to include this service for users since it takes a 1mb file and turns it into a 1gb file! We have a meeting tomorrow to discuss some alternatives- but did you try using grads-dods to access in NetCDF? Please see COLA docs for info. I'll let you know what we come up with on this. Glenn Jon Goodall wrote: Hi Glenn, I wanted to follow-up on the NARR in netCDF conversion you were working on a few weeks ago. How's it going? Thanks, Jon Goodall Center for Research in Water Resources University of Texas at Austin ________________________________ Subject: Re: NCEP North American Reanalysis (fwd) From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> Date: Tue, 12 Oct 2004 09:46:05 -0400 To: goodalljl@xxxxxxxxxxxxxxx Hi Jon, We just developed the capability to do this and find that it will not be wise for me to include this service for users since it takes a 1mb file and turns it into a 1gb file! We have a meeting tomorrow to discuss some alternatives- but did you try using grads-dods to access in NetCDF? Please see COLA docs for info. I'll let you know what we come up with on this. Glenn Jon Goodall wrote: Hi Glenn, I wanted to follow-up on the NARR in netCDF conversion you were working on a few weeks ago. How's it going? Thanks, Jon Goodall Center for Research in Water Resources University of Texas at Austin ----Original Message Follows---- From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> To: goodalljl@xxxxxxxxxxxxxxx CC: worley@xxxxxxxx, ben@xxxxxxxxxxxxxxxx, maidment@xxxxxxxxxxxxxxx, dennisl@xxxxxxxxxxxxxxxx, support-ldm@xxxxxxxxxxxxxxxx, benno@xxxxxxxxxxxxxxxxxxxxx, Don.K.Hooper@xxxxxxxx, roland.schweitzer@xxxxxxxx, rkambic@xxxxxxxxxxxxxxxx, Sharon.Leduc@xxxxxxxx, Russell.Vose@xxxxxxxx, Dan.Swank@xxxxxxxx Subject: Re: NCEP North American Reanalysis (fwd) Date: Fri, 01 Oct 2004 10:28:12 -0400 ________________________________ Subject: Re: NCEP North American Reanalysis (fwd) From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> Date: Fri, 01 Oct 2004 10:28:12 -0400 To: goodalljl@xxxxxxxxxxxxxxx Thanks Jon, I stand corrected- it does work, but for a limited volume only (as in your example!). The conversion is rather straightforward, and w/o issues it could be by next week. Glenn Jon Goodall wrote: Hi All, I've been able to download a subset of the NARR data (only surface-level values from a week at the 3 hour time step) from the NCDC website using the ftp4u link. It works great and I had no problems. If we also have the grib2nc conversion, we will be all set. Glenn, when do you hope to has this conversion available? Jon Goodall Center for Research in Water Resources University of Texas at Austin ----Original Message Follows---- From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> To: Steve Worley <worley@xxxxxxxx> CC: Ben Domenico <ben@xxxxxxxxxxxxxxxx>, David Maidment <maidment@xxxxxxxxxxxxxxx>, Jon Goodall <goodalljl@xxxxxxxxxxx>, "Dennis P. Lettenmaier" <dennisl@xxxxxxxxxxxxxxxx>, support-ldm <support-ldm@xxxxxxxxxxxxxxxx>, Martin Benno Blumenthal <benno@xxxxxxxxxxxxxxxxxxxxx>, Don Hooper <Don.K.Hooper@xxxxxxxx>, Roland Schweitzer <roland.schweitzer@xxxxxxxx>, Robb Kambic <rkambic@xxxxxxxxxxxxxxxx>, Sharon Leduc <Sharon.Leduc@xxxxxxxx>, Russell Vose <Russell.Vose@xxxxxxxx>, Dan Swank <Dan.Swank@xxxxxxxx> Subject: Re: NCEP North American Reanalysis (fwd) Date: Fri, 01 Oct 2004 09:04:49 -0400 ________________________________ Subject: Re: NCEP North American Reanalysis (fwd) From: "Glenn Rutledge" <Glenn.Rutledge@xxxxxxxx> Date: Fri, 01 Oct 2004 09:04:49 -0400 To: Steve Worley <worley@xxxxxxxx> Thanks Steve and Ben for your forward. We are now ingesting and have avbl the NARR. We have 1987 thru 2003 (of the 1979-2004) as requested below. NCAR and SDSC will also have copies. Here at NCDC, Murphy got in the way as I usually have grib sub-seting capabilities, however the RAID I expected in for this operation has not yet arrived! I'm told the check is in the mail. To get around this, the NOMADS team (all 2.5 of us) put together direct access http to (m)get at the data due to this hardware delay. So, users may access the data as per Steve's url below. Use the http link to get at the data directories. Depending on your internet connection, it may be a slow process (maybe open 2 ports). By next week (or so), please point users to the "ftp4u" capability that can subset this large (5TB) dataset by parameter space and time (and by variable). There is dataset documentation also avbl at the site. As I write, we are developing a grib2nc capability for users to retrieve in NetCDF format. We will also compress the NetCDF for transmission (GFDL, Kerr) found that these NetCDF files compress very well. Please let me know your experiences with this site and do try to subset when that's avbl. Thanks, Glenn Steve Worley wrote: All, North American Regional Reanalysis is just coming available from two location I know of: the NCDC NOMADS server (Glenn's work) at http://nomads.ncdc.noaa.gov/data-access.html#narr_datasets and NCAR through the Data Support Section. We have 1987-2003 archived, but not fully ready for service. However, if you wish to let us know what you require we might be able to accomodate your needs without too much problem. Chi-Fan Shih (chifan@xxxxxxxx) can address your further questions about NARR from NCAR. sjw Ben Domenico wrote: Hi David, Jon and Dennis, Your questions are flattering in terms of the assumed breadth and depth of knowledge about data collections here at the Unidata Program Center (UPC), but we should be honest and offer a few clarifications that will help you find what you are looking for more quickly. The main clarification is that the staff at the UPC mainly focuses on infrastructure for real-time data systems for weather data. I understand that archived climate information is closely related, but your questions about the details of specific archived climate datasets are skirting pretty much at and beyond the edges of our UPC expertise. Those sorts of questions are better posed to the folk who create and maintain the climate data archives at places like the NCDC, NCAR, Lamont-Doherty, and CDC. I plan to send a copy of this note to Glenn Rutledge at NCDC, Benno Blumenthal at LDEO, Steve Worley at NCAR, and Don Hooper & Roland Schweitzer at CDC who should be able to help you find what you need in terms of the North American Reanalysis datasets -- in particular a netCDF version. On the other hand, we do have experience at the UPC with decoders that transform certain GRIB datasets into netCDF form (indeed with considerably expanded size) and I'll ask Robb to follow up on that issue to see if there is a version of that (GRIBTONC) decoder for the North American Reanalysis data. I don't mean to be passing the buck, but there are many groups who have more detailed expertise than the UPC about archives of climate datasets. --Ben ****************************************************** Ben Domenico Unidata Deputy Director http://my.unidata.ucar.edu P.O. Box 3000 ben@xxxxxxxxxxxxxxxx Boulder, CO 80307 (303)497-8631 FAX: (303)497-8690 --On Thursday, September 30, 2004 3:45 PM -0500 David Maidment <maidment@xxxxxxxxxxxxxxx> wrote: Robb: Dennis Lettenmaier is correct here. What you are pointing us to is the global NCEP reanalysis, which is on a 2.5 degree grid and not very informative for the hydrology of the US. What we are seeking to work with is the reanalysis for North America alone, which is on a 32km grid and is in Grib format, not netCDF. The global reanalysis is available in netCDF. David -----Original Message----- From: Robb Kambic [mailto:rkambic@xxxxxxxxxxxxxxxxxxxxxxxx] Sent: Thursday, September 30, 2004 11:25 AM To: Ben Domenico Cc: David Maidment; support-ldm Subject: Re: NCEP North American Reanalysis (fwd) look at url: http://www.cdc.noaa.gov/cdc/data.ncep.reanalysis.html#surface_gauss parameters water runoff, soil moisture, etc. these files are in netcdf so they should be usuable for your analysis. i downloaded a couple of files and dumped them using ncdump. there are proably other hydrologic fluxes data sets that can be found by searching the web. robb... On Thu, 30 Sep 2004, Ben Domenico wrote: Hi all, The note below from David Maidment asks about getting the NCAR reanalysis products into netCDF form. Is there something special about these datasets or would our grib to netcdf decoder work with them. Regarding the grads part of it, I'm assuming that refers to the fact that they are available via the GDS protocol, but let me know if there's more to it than that. And how about the CDC archives? I thought they published these data in netCDF form. Thanks. -- Ben ---------- Forwarded Message ---------- Date: Thursday, September 30, 2004 8:18 AM -0500 From: David Maidment <maidment@xxxxxxxxxxxxxxx> To: Ben Domenico <ben@xxxxxxxxxxxxxxxx> Subject: NCEP North American Reanalysis Ben: In CUAHSI HIS, we are thinking that a good "coverage" of hydrologic fluxes from the atmosphere could be obtained from the NCEP North American Reanalysis. More info is at http://www.emc.ncep.noaa.gov/mmb/rreanl/ It seems like the data are all stored in one of your sister institutions in Boulder. Could you please help us to interpret what the data available are? The data are in Grib/Grads format. We have invested a lot of effort in working with netCDF and I don't want Jon Goodall or others in our CUAHSI community having to tussle with Grib/Grads. We have another other data formats already with GIS, point observations and remote sensing to consider. Would it be possible to translate these reanalysis data layers into netCDF format? David ---------- End Forwarded Message ---------- ======================================================================= ====== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ ======================================================================= ======
============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ ==============================================================================
decoders archives: