Thanks for the quick reply, Tom. Looking through our conduit2 logs, we
began seeing sends of product from our conduit2 to conduit1 machine after
we restarted the LDM server on conduit2 yesterday. It appears latencies
improved fairly significantly at that time:
[image: image.png]
However we still do not see direct sends from conduit2 to external LDMs.
Our server team is currently looking into the TCP service issue that
appears to be causing this problem.
Thanks,
Jesse
On Mon, Aug 30, 2021 at 7:49 PM Tom Yoksas <yoksas@xxxxxxxx> wrote:
> Hi Jesse,
>
> On 8/30/21 5:16 PM, Jesse Marks - NOAA Affiliate wrote:
> > Quick question: how are you computing these latencies?
>
> Latency in the LDM/IDD context is the time difference between when a
> product is first put into an LDM queue for redistribution and the time
> it is received by a downstream machine. This measure of latency, of
> course, requires that the clocks on the originating and receiving
> machines be maintained accurately.
>
> re:
> > More
> > specifically, how do you determine which conduit machine the data is
> > coming from?
>
> The machine on which the product is inserted into the LDM queue is
> available in the LDM transaction. We provide an website where users
> can create graphs of things like feed latencies:
>
> Unidata HomePage
> https://www.unidata.ucar.edu
>
> IDD Operational Status
> https://rtstats.unidata.ucar.edu/rtstats/
>
> Real-time IDD Statistics -> Statistics by Host
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex
>
> The variety of measures of feed quality for the Unidata machine that
> is REQUESTing the CNODUIT feed from the NCEP cluster can be found in:
>
>
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu
>
> The latencies being reported by the Unidata machine that is being fed
> from the NCEP cluster is:
>
> CONDUIT latencies on conduit.conduit.unidata.ucar.edu:
>
>
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>
> As you can see, the traces are color color coded, and the label at the
> top identifies the source machines for products.
>
> re:
> > The reason I ask is because I am not seeing any sends of
> > product from conduit2 in the last several days of logs both to our local
> > conduit1 machine and to any distant end users.
>
> Hmm... we are.
>
> re:
> > Also, we have isolated what is likely the issue and will have our team
> > take a closer look in the morning. I'm hopeful they'll be able to
> > resolve this soon.
>
> Excellent! We are hopeful that the source of the high latencies will
> be identified and fixed.
>
> Cheers,
>
> Tom
>
> > On Mon, Aug 30, 2021 at 5:24 PM Anne Myckow - NOAA Federal
> > <anne.myckow@xxxxxxxx <mailto:anne.myckow@xxxxxxxx>> wrote:
> >
> > Pete,
> >
> > Random aside, can you please update your doco to say that
> > Dataflow's email list is now nco.dataflow@xxxxxxxx
> > <mailto:nco.dataflow@xxxxxxxx> ? I'm CC'ing it here. That other
> > email address is going to get turned off within the next year.
> >
> > Thanks,
> > Anne
> >
> > On Wed, Aug 18, 2021 at 4:02 PM Pete Pokrandt <poker@xxxxxxxxxxxx
> > <mailto:poker@xxxxxxxxxxxx>> wrote:
> >
> > Dear Anne, Dustin and all,
> >
> > Recently we have noticed fairly high latencies on the CONDUIT
> > ldm data feed originating from the machine
> > vm-lnx-conduit2.ncep.noaa.gov
> > <http://vm-lnx-conduit2.ncep.noaa.gov>. The feed originating
> > from vm-lnx-conduit1.ncep.noaa.gov
> > <http://vm-lnx-conduit1.ncep.noaa.gov> does not have the high
> > latencies. Unidata and other top level feeds are seeing similar
> > high latencies from vm-lnx-conduit2.ncep.noaa.gov
> > <http://vm-lnx-conduit2.ncep.noaa.gov>.
> >
> > Here are some graphs showing the latencies that I'm seeing:
> >
> > From
> >
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
> > <
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
> > -
> > latencies for CONDUIT data arriving at our UW-Madison AOS ingest
> > machine
> >
> >
> >
> > From
> >
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu
> > <
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu
> > (latencies
> > at Unidata)
> >
> >
> >
> > At least here at UW-Madison, these latencies are causing us to
> > lose some data during the large GFS/GEFS periods.
> >
> > Any idea what might be causing this?
> >
> > Pete
> >
> >
> >
> >
> > <
> http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool
> >-----
> > Pete Pokrandt - Systems Programmer
> > UW-Madison Dept of Atmospheric and Oceanic Sciences
> > 608-262-3086 - poker@xxxxxxxxxxxx <mailto:poker@xxxxxxxxxxxx>
> >
> >
> >
> > --
> > Anne Myckow
> > Dataflow Team Lead
> > NWS/NCEP/NCO
> >
> >
> >
> > --
> > Jesse Marks
> > Dataflow Analyst
> > NCEP Central Operations
> > 678-896-9420
>
> --
> +----------------------------------------------------------------------+
> * Tom Yoksas UCAR Unidata Program *
> * (303) 497-8642 (last resort) P.O. Box 3000 *
> * yoksas@xxxxxxxx Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------------+
>
--
Jesse Marks
Dataflow Analyst
NCEP Central Operations
678-896-9420
