[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[McIDAS #DIB-989996]: XCD future

Hi Jerry,

I apologize for the slow response.  I received your message while in
Phoenix at the AMS Annual meeting.  I have been working on some other
things since I returned, and only now have had some time to respond...

re:
> I was hoping to get your input on some ideas we have floating around
> about XCD.
> 
> The Data Center and a few of the MUG folks were talking about the
> future of XCD.   I had held a belief that much of what XCD does is over
> complicated, and better served by a rewrite or at least a rethinking of
> how things are done.

I agree completely about some of the setup being overly complicated.

> It seems that LDM in combination with a few new data decoders set up in
> pqact.conf can replace much of XCD.

This would be a great improvement in how things work!

> I was wondering your feelings are about this?

I'm in!!!

> What in XCD do you
> folks use?

We use:

- decoding of NOAAPort-ingested observational data into MDXX files

- decoding of other text messages into TEXT ADDE datasets

- GRID decoding and GRIB serving (as soon as the bugs are fixed
  in the GRIB serving, I will drop the GRID decoding like a hot
  potato!)

> What decoders do you feel would be required in a
> replacement?

Generalized versions of decoders for POINT data, TEXT data, and GRIB
serving.

> This is at the "just a kicking around some ideas" stage, since no
> coding has been done, and no funding has been allocated.

I understand.  I would love to see XCD modularized to the point that
the user could use the decoders directly from the LDM or "by-hand"
from the *nix command line.

> Here are some of my ideas,
> 
> Text  data  -  port current data decoder that creates a daily text
> file, and files meta data in pointer files (similar to now.) end
> product would be the same as now.

Sounds good.

> Point/MD data  -  port MD decoder from XCD and create MD files the same
> as XCD but from a decoder (combine DMMISC, DMRAOB, DMSFC, and DMSYN
> into one decoder, use config file to determine which data types to
> decode)

Also sounds good.  Support for decoding of historic datasets would be
a BIG win for the users.

> Nexrad - just use LDM .... no decoder needed, unless we need a simple
> decoder to manage the directory structure

This is what we do now.  We have never used the DMNEXR piece of XCD.

> Grib data - write a decoder that extracts meta data, and files it into
> an SQL database.  Similar to what we have now, but maybe use SQL lite.

Excellent.  Just so you are aware:  there are some bugs in the GRIB processing
code that do not come to light on RedHat or its variant (e.g., Fedora) Linux
systems.  They do manifest themselves, however, on Solaris x86 especially
in 64-bit mode.  I have made a number of changes to GRIB-related routines
in Unidata McIDAS, but I have not found the final ones that allow the GRIB
processing to work without error.  It is my intention to update the XCD CVS
at SSEC when I do have a working solution.

> Bufr data - no decoder needed file directly.

OK assuming that BUFR serving is fully implemented.

> GRID data - not supported

I'm with you on this one!

> TIROS -  Modify Tiros decoder to file SYSNAV similar to what is currently 
> done.

OK.

> I see 4, maybe 5, decoders called directly from LDM.  Each would have a
> configuration file for site specific information.

Sounds good.  This would make Unidata user's lives easier as that is basically
how all other decoders work.

> Thoughts on how would this be impacted by the next generation ldm?

This is unknown right now, but conceptual compatibility is one of our top
goals.

> What is the "java" ldm version timeline?

Good question.  Steve is working as fast/hard as he can.  The job is large
and the impact will be so huge that we want to get this right before it
is made available in any form.

> Your thoughts on this is appreciated.

I am 100% for modification of XCD in the manner that you outlined.  I think
that it will make end-users jobs easier and be easier to maintain.

By the way.  The "Unidata way" of configuring XCD is significantly different
than the SSEC way.  As you know, I bundle -X and -XCD into our distribution.
Both packages are built in the 'mcidas' account.  The decoding is run from
the LDM account, not from an 'oper' account.

The LDM setup is composed of three pieces:

1) a Bourne shell script, xcd_run, that handles details like setting
   of needed McIDAS environment variables, etc.  I have attached a
   version of xcd_run to this reply

2) ~ldm/etc/ldmd.conf entries that process the data:

#
# McIDAS-XCD entries
#
exec    "xcd_run MONITOR"
exec    "pqact -f IDS|DDPLUS|HRS|NGRID|FNEXRAD|NIMAGE etc/pqact.conf_mcidasA"
exec    "pqact -f FSL2|NLDN|NEXRAD3|UNIWISC etc/pqact.conf_mcidasB"

3) appropriate actions in McIDAS-specific pqact.conf file(s):

#########################################################################
#
# McIDAS-XCD section using 'xcd_run' wrapper for POINT type data
#
# NOTEs:
#        - copy 'xcd_run' from ~mcidas/bin directory to ~ldm/decoders 
#        - edit 'xcd_run' and review settings
#        - make sure that ~ldm/decoders is in the PATH for 'ldm'
#        - stop and restart the LDM to activate
#
#########################################################################

# NOAAPORT-broadcast observational data
IDS|DDPLUS      .*      PIPE
        xcd_run DDS

#########################################################################
#
# McIDAS-XCD section using 'xcd_run' wrapper for GRIB data
#
# NOTEs:
#        - copy 'xcd_run' from ~mcidas/bin directory to ~ldm/decoders 
#        - edit 'xcd_run' and review settings
#        - make sure that ~ldm/decoders is in the PATH for 'ldm'
#        - stop and restart the LDM to activate
#
#########################################################################

# NOAAPORT-broadcast model output
HRS|NGRID       .*      PIPE
        xcd_run HRS


Required on the user's side is:

1) 'ldm' and 'mcidas' (and 'mcadde') are put into the same group
2) the umask for 'ldm' and 'mcidas' are set so each can read/write
   each other's files/directories
3) the ~mcidas/bin directory has read/write/execute permission for
   the user 'ldm'
4) the ~mcidas/workdata directory (the working directory for Unidata
   'mcidas' users) is set to be read/write/execute for the user 'ldm'

All McIDAS-related configurations are handled in 'xcd_run' and in
the 'mcidas' account.  The 'ldm' account does not have to know about
McIDAS stuff (e.g., environment variables, etc.).

The good thing about this approach is that it is a LOT simpler for
the end user.  The bad thing about this approach is that the 'ldm'
and 'mcidas' accounts need to be somewhat connected to one another
since they each can write in each other's directories.

The other thing that is different in the Unidata McIDAS distribution
is the setup for ADDE serving.  Unidata McIDAS requires the existence
of the user 'mcadde', but the HOME directory for this user is the same
as the user 'mcidas'.  The only difference in the accounts is that
'mcadde' is not a login account.  'mcidas', on the other hand' is a login
account AND McIDAS can be run from the 'mcidas' account (in SSEC McIDAS,
the user is prevented from running McIDAS through a trap in the startup
script "mcidas".  This is one of the first things I change with every
McIDAS release.

I view the setup for Unidata users to be simpler since all configurations
are done through one account, 'mcidas'.  There is no 'oper' user, and
'mcadde' is in large part a pseudonym for 'mcidas'.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: DIB-989996
Department: Support McIDAS
Priority: Normal
Status: Closed