[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #IJN-200350]: 20060708: EAST and WEST datasets not updating on unidata2

Hi Jerry,

re:
> The ldm relay of gvar data has been running since Friday ... so far, after 
> the permission problem
> got fixed Friday morning, it seems to be working well.

I agree.

> Have you had a chance to evaluate the
> impact on unidata2.ssec.wisc.edu?  Should the queue remain the current size?

Mike Schmidt and I chatted about the impact on unidata2's LDM, which is a 
shorter set of realtime
data available in the queue.  We discussed two approaches for mitigating the 
impact:

- increasing the size of the queue from 2 GB to 3 GB

- assigning a second, virtual IP address to unidata2 and running a second LDM 
that listens
  only to this address

The first option is quick and easy to implement, but I wanted to review the 
processing
you wrote for the GOES data to make sure that resends of the same data would not
harm the ADDE serving of EAST/WEST data.

Last night, Mike suggested the second approach as being more attractive:

  Date: Mon, 24 Jul 2006 18:34:33 -0600)
  To: address@hidden
  Subject: moving goes data to unidata2

  Tom,

  After very little thought, I think we should consider moving GOES data
  to unidata2 in the fashion we were discussing for motherlode (via a
  virtual IP and binding a second LDM to that address).  This would avoid
  over-committing memory resources needed for the operational LDM service,
  especially on unidata2 where there's currently no expandability and
  currently no free memory.

  We'd have to request another IP address from SSEC to make it go.

  mike

> Currently we have two independent systems ingesting both operational GVAR 
> streams(gserve1 and
> gserve2).   I only have one of the systems relaying the data via the ldm to 
> unidata2.   It would be
> nice if both systems were set up to relay the data, so if one system goes 
> down, or is taken down for
> maintenance, the other system will still be supplying the data.  Do you think 
> it be possible for
> both systems to relay the data without consuming too much bandwidth?

I don't think that bandwidth is much of an issue for the Data Center.  That 
being said, LDM 6
allows one to setup redundant feed requests and only one, the fastest one, will 
result in files
actually being transferred.  This is the approach that I would recommend 
_after_ bringing up
a Virtual interface and running a second LDM.

> Also, I can think of a couple of problems with doing this ... while the data 
> between the systems is
> nearly identical .... a few bits of gvar data may be different on occasion 
> between the two systems
> .. not very many, but both ingestors are completely independent, and unless 
> they are started at
> precisely the same moment, there sometimes are a bit or two different on a 
> couple images a day.
> This would mean that the checksums for the same image will be different on 
> the two servers.  The ldm
> would treat these as different, and pull both over to unidata2 ... correct?

Correct.  The duplicate product detection and rejection only works when the MD5 
checksums are identical.

> And this would use
> additional bandwidth, and allow for less products to remain in the queue?

Yes and yes.

> Any ideas, or should we just use one system for now?

We would like to request a second IP address for unidata2.  We would then like 
to bring up
a separate LDM listening only to that interface and dedicated for the GOES data 
transfers.
After accomplishing that, we will be in a position to think about a different 
approach to
requesting the data redundantly.  One idea that comes to mind immediately 
(unencumbered
by the thought process ;-) is calculating the MD5 signature in a different 
manner, like
from meta data that is available.  By not assigning the signature the value of 
the data
product checksum, we could insure that the signatures from different machines 
would be
the same.  This would allow the LDM to reject one of the redundantly requested 
products.
I like this approach (for now) because we do not know apriori which of the 
images that
are different is "better".

> Thanks for any ideas.

Please let us know what you think the issues above (i.e., second, virtual 
interface;
second LDM; different manner of calculating MD5 signature).

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: IJN-200350
Department: Support Datastream
Priority: Low
Status: Closed