[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[SCOOP #OCU-940339]: Topics for discussion...



Hi Gerry,

re: 
> I'm in San Diego, and I think you're at AGU, this week,

I and neither of the Steves (Chiz and Steve Emmerson) will be at the AGU
meeting in San Francisco.  I believe that "the Steves" will be in Boulder;
I will be in the Netherlands attending a meeting that is scheduled for
this coming Thursday and Friday (although I leave tomorrow and don't
return until Sunday).

> but perhaps over
> the next couple of days we could visit via e-mail about a transport
> monitor issue we're seeing in SCOOP.

If the reference to the .txt file below is the transport monitor issue,
then a simple solution would be the addition of a timestamp field in
the file before using pqinsert to insert it into the queue -- anything
that would change the MD5 signature would make the product unique.  We
added a single byte to the end of NEXRAD Level III products that were
sent out as floater sites in the FNEXRAD feed even while they were being
sent in the NNEXRAD feed from the same machine.

Another approach would be to create your own custom pqinsert that creates
an MD5 signature using the body of the product (.txt file in this case)
and the current time.

> I also kicked off the about gathering the SCOOP LDM admins at TAMU on 5
> MAR, one day prior to our Spring "All Hands" meeting.  Would one, or
> both of you be willing to come visit with us on 5 MAR to discuss our use
> and implementation of a SCOOP IDD and to help tweak and clean things up?

I believe that one of us could attend the LDM admins meeting (the trip will
have to be approved, of course).  I think that Chiz could bring the most
authoritative knowledge to the table given his leading role in CONDUIT.

I will touch base with Chiz to see if he could be available to participate in
the LDM admins meeting.

Also, pqinsert was modified in the LDM-6.4.5 distribution to return some
status other than zero when attempting to insert a duplicate product in the
LDM queue.  Here is the snippit regarding the change in the CHANGE_LOG
file in the LDM distribution:

        Modified pqinsert(1) so that it exits with a non-zero status if
        something went wrong inserting a file into the product-queue.

This would indicate that the site(s) doing the pqinsert(s) should upgrade to
a current release of the LDM in order to be able to use pqinsert's exit status
code in shell scripts.

Finally, I need to ask the following:

- who would pay for the trip to the LDM admins meeting

- would the meeting be structured so that recommended changes could be
  affected immediately (i.e., make it a working meeting whose goal would
  be to revamp the distribution setup then and there)?  If yes, more time
  might need to be dedicated to the portion of the meeting aimed at
  implementing the changes.

> -------- Original Message --------
> Subject: Re: [Scoop] Re: Transport Monitor (SWW3LLMF-BIO-WANAFe??-UFL)
> Date: Wed, 6 Dec 2006 14:34:52 -0600
> From: Jon MacLaren <address@hidden>
> To: Donna Cote <address@hidden>
> CC: Smith, Matt <address@hidden>,     Maskey, Manil
> <address@hidden>,     scoop team at LSU <address@hidden>,     Keiser,
> Ken <address@hidden>, Gerry Creager <address@hidden>
> References: <address@hidden>
> <address@hidden>
> 
> Hi Donna, Matt, and everyone else,
> 
> The problem is the same as before.  Identical .txt files, which LDM
> thinks are already in the queue.  pqinsert returns zero whether it
> succeeds or not.  There is no real fix for this, that we can make.
> 
> When we first discussed this (13th Nov), I suggested a "workaround"
> of sending the .txt files out compressed, which works, because the
> gzip always results in a different MD5 checksum, even for identical
> files because it encodes the filename in the archive, and in our case
> these are always different.  (This approach has been used in other
> places, I think by Justin.)  I offered to implement the above
> solution.  No-one responded saying yes, so I didn't.
> 
> But also, the files are not important, because we're supposed to be
> using the .nc files now, which don't have this problem.
> 
> Your call Donna - I can either stop sending the files, because
> they're obsolete, or I can gzip them so pqinsert works.  But these
> are the only two fixes on the table.
> 
> We need to push Unidata to fix pqinsert too.  Gerry, can you help
> with that?
> 
> Jon.
> 
> 
> On Dec 5, 2006, at 10:33 PM, Donna Cote wrote:
> 
> > Matt,
> >
> > From what I have found, we are receiving all of the ANA-driven WW3
> > files from LSU that LSU sends. When this happened before, Jon
> > complained that his script reported to the transportLog that
> > the .txt files were "generated" but the pqinsert failed ("already
> > in queue" message). Actually, Jon's complaint was that he had no
> > way to know whether the pqinsert ran or failed. I was going to
> > experiment a bit with that and didn't get 'round tuit'. Let me work
> > with Jon, or his rep of the week, and see if we can't get this
> > fixed on their end.
> >
> > Donna
> >
> > Some details:
> > $ grep SWW3LLMF-BIO_WANAFe ldmd.log.2 | grep
> > UFL_20050827T0000_20050827T0000_20050827T | grep T342 | cat > /tmp/
> > ww3T342-ldmlist
> > $ wc -l /tmp/ww3T342-ldmlist
> > 198
> > $ grep -v scoop-queuing /tmp/ww3T342-ldmlist | grep -v rtstats |
> > grep Z.txt | wc -l
> > 33
> > $ grep -v scoop-queuing /tmp/ww3T342-ldmlist | grep -v rtstats |
> > grep Z.nc.gz | wc -l
> > 33
> > (for a total of 66) but, for T345:
> > $ grep SWW3LLMF-BIO_WANAFe ldmd.log | grep
> > UFL_20050827T0000_20050827T0000_20050827T | grep T345 | cat > /tmp/
> > ww3T345-ldmlist
> > $ wc -l /tmp/ww3T345-ldmlist
> > 99
> > $ grep -v scoop-queuing /tmp/ww3T345-ldmlist | grep -v rtstats |
> > grep Z.txt | wc -l
> > 0
> > $ grep -v scoop-queuing /tmp/ww3T345-ldmlist | grep -v rtstats |
> > grep Z.nc.gz | wc -l
> > 33
> > (for a total of 33)
> >
> > Smith, Matt wrote:
> >>
> >> Donna,
> >>
> >> You&#146;re only receiving (or at least only telling us about) 33 ANA-
> >> driven WW3 files from LSU &#150; not 66. Looks like you&#146;re still not
> >> getting (or telling us about) the text versions of the same 33
> >> files. Do you want to continue looking into it &#150; or shall I simply
> >> change TAMU&#146;s expected filecount to 33?
> >>
> >> Thanks,
> >>
> >> Matt
> >>
> >> //Matt//// Smith ///
> >> //Information//// Technology & Systems Center //
> >> //University//// of Alabama in Huntsville //
> >> //at the National Space Science and Technology Center //
> >> //320 Sparkman Dr////. //
> >> //Huntsville////, AL 35805//// //
> >> //ph 256-961-7809 fax 256-961-7859 //
> >> //e-mail /////address@hidden// <mailto:address@hidden>
> >>
> > --
> > Donna Cote
> > Senior Research Associate
> > The Academy for Advanced Telecommunications and Learning Technologies
> > Texas A&M University
> > 3139 TAMU
> > College Station, Texas 77843-3139
> > Office: (979) 862-3982
> > Cell: (979) 324-3549
> > Fax: (979) 862-3983
> >
> > _______________________________________________
> > Scoop mailing list
> > address@hidden
> > https://mail.cct.lsu.edu/mailman/listinfo/scoop
> >
> 
> 
> --
> Gerry Creager -- address@hidden
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
> 
> 

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: OCU-940339
Department: Support IDD SCOOP
Priority: Normal
Status: Open