[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #ZRG-608224]: SSEC neartime data archive



Hi Dave,

re: switch of source point for Unidata-Wisconsin and FNEXRAD IDD datastreams
> I think I needed to have been reading my Unidata newsletters more carefully
> —I didn’t see that coming. :-)

We announced the change in the idvusers, gembud, ldm-users and mcidas-x
email lists.  I am surprised that you didn't get the info in at least
one of these.

re: 
> Yes. My local LDM died for some reason for about 6 hours today (March 25).

OK.  Backfilling real-time datasets populated by data received in the
IDD and grabbing sets of old data for case studies is exactly what
the Unidata-Wisconsin archives was designed for.

re: multi day (something like 30) rolling archives of Unidata-Wisconsin
data are available on the motherlode class machines we operate, and they
are served by ADDE

> I don’t actually know how to save data files accessed by the IDV (except
> as part of a *.zidv file). Came up empty searching the Users Guide about
> how to do this, too.

The IDV has an interface to save data that has been loaded to a netCDF file.
The IDV guys know a lot more about this than I do, so I will not try to
describe how this is done here.

re:
(I need to save files in a format that WXP program “xsat” can read—MCIDAS
> AREA files, in particular.)

Saving in McIDAS AREA format is _not_ supported in the IDV ... I wish that
it was!

re: UNIWISC directory trees accessible through the web
 
> I downloaded a couple of these files from motherlode (one from
> "decoded > gempak > areas", and one from "mcidas > RTIMAGES > GW-IR”.
> The file from the latter location is the same size as the MCIDAS
> AREA files that we acquire using LDM/IDD.

Actually, the files grabbed from the decoded->gempak->areas subdirectories
are the same size as delivered in the LDM/IDD since the images are simply
FILEd by an LDM pattern-action file action.  The difference in size is
like to be caused by you running the files received in the LDM/IDD through
the ldm-mcidas 'pnga2area' decoder.  You will likely need to do the same
thing with the files you snag via the web in order for them to be usable
by WXP (GEMPAK and McIDAS-X can use the files directly, and I thought
that the IDV could do the same, but I am not 100% sure).

re:
> However, I couldn’t read either of them using the IDV. (I tried specifying
> data source types “MCIDAS AREA files” and “Image Files (GINI)”, among other,
> less promising possibilities).

I am surprised by this.  I will run a couple of tests with my IDV to verify
that the IDV can or cannot read the PNG-compressed images (NB: only the
data portion of the image is PNG compressed; the header is uncompressed,
so you can not run the images through a generic PNG decompression routine,
you have to use the ldm-mcidas 'pnga2area' decoders).

re:
> I was able to read the MCIDAS AREA file downloaded from directory
> "mcidas > RTIMAGES > GW-IR "using WXP’s “xsat” program, though,

This images are created by running the PNG-compressed products through
'pnga2area'.

re:
> and that’s actually what I need in principle (for auto-generating products
> for the California Regional Weather Server).

If you use 'pnga2area' on the images you snag via the web interface
discussed above, the resulting images will be in pristine McIDAS AREA
format.

re:
> Looks like only the most recent 10 images are stored in each subdirectory
> of "mcidas > RTIMAGES", though,

Correct.

re:
> and I haven’t completely figured out the naming convention yet. For example,
> in "mcidas > RTIMAGES > GW-IR”,  file “AREA0130” is the youngest, “AREA0139”
> the next youngest, and “AREA0131” the oldest, but that pattern doesn’t hold
> for images in other subdirectories, suggesting that the last digit (0-9)
> rotates as new images arrive. Automating the acquisition of missing images
> from this location would require some brain cells to program …. :-)

Your sleuthing is right on the money: the sets you are referring to are
best thought of as circular spools of names.  As new images of a type
come in and are processed, the oldest in a set is replaced the the
newest.  The only way to know what the time is in the file is by reading
the AREA directory and extracting the date/time.

re:
> How will I find out when the Amazon cloud archive becomes operational,

It is not our intention to host the Unidata-Wisconsin archive in the cloud,
at least not yet.  The reason for this is that heavy use of the archive
(meaning transferring a lot of images out of it) would not be affordable.
The reason for this is that network traffic out of Amazon EC2 is charged
separately from having and using VMs in that cloud.  Mike Schmidt did
a calculation about a year ago (or was it 6 months) that indicted that
it would cost on the order of a quarter of a million dollars per year
to run our motherlode server in EC2.  The biggest portion of this cost
was the network traffic being sent out of the cloud; it appears that
moving data into the cloud is cheap and moving data withing the same
cloud may, in fact, be free (at least at the present time).

re:
> and how to access files there? (I have scripts that run automatically
> to download images from SSEC when we don’t get them via LDM/IDD—I’ll
> need to rejigger those scripts accordingly.)

It is likely that we will move the Unidata-Wisconsin archive to either
another machine in the UW/SSEC Data Center or to a machine here in
the UPC.  Which approach we take will be governed by the cost (the
Unidata subaward to SSEC was cut by about 29% in the latest budget,
and this is having repercussions on what can be done where).

Please try running the images snagged from the decoded->gempak->areas
subdirectories through 'pnga2area' and see if they are not fully
compatible with your current workflows.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZRG-608224
Department: Support Datastream
Priority: Normal
Status: Closed