Keep providing the feedback. All of it is being given to the NOAA NOC
admins. This seems to be a pretty wide spread issue with numerous circuits
being impacted. As soon as I have more information about resolution I will
pass it on.
Carissa Klemmer
NCEP Central Operations
Dataflow Team Lead
301-683-3835
On Mon, Feb 22, 2016 at 1:32 PM, <admin@xxxxxxxx> wrote:
> Those traceroutes clearly show the issue is on the last 2 hops or inside
> NOAA.
>
> Ray
>
> On Friday, February 19, 2016 2:39pm, "Gerry Creager - NOAA Affiliate" <
> gerry.creager@xxxxxxxx> said:
>
> > _______________________________________________
> > conduit mailing list
> > conduit@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/ If I were the conspiracy
> theory type, I
> > might think to blame the
> > balkanization of all paths to the internet from NOAA sites via the
> Trusted
> > Internet Connection stuff. But I'm just hypothesizing. Or, they could be
> > running distro on overloaded VMs.
> >
> > gerry
> >
> > On Fri, Feb 19, 2016 at 1:28 PM, Patrick L. Francis <
> wxprofessor@xxxxxxxxx>
> > wrote:
> >
> >>
> >>
> >> Art / Pete etc. al. J
> >>
> >>
> >>
> >> There seems to be a consistency is potential packet loss from no matter
> >> which route is taken into ncep… so whoever you are communicating with,
> >> you
> >> might have them investigate 140.90.111.36… reference the previous
> graphic
> >> shown and this new one here:
> >>
> >>
> >>
> >> http://drmalachi.org/files/ncep/ec2-ncep.png
> >>
> >>
> >>
> >> if you are unfamiliar with amazon ec2 routing, the first.. twenty
> >> something or so hops are just internal to amazon, and they don’t jump
> >> outside until you hit the internet2 hops, which then jump to gigapop,
> and
> >> from there to noaa internal.. so since this amazon box is in ashburn,
> >> physically it’s close, and has limited interruptions until that point..
> >>
> >>
> >>
> >> the same hop causes more severe problems from my colo boxes, which are
> >> hurricane electric direct, which means that in those cases jumping from
> >> hurricane electric to 140.90.111.36 has “severe” problems (including
> >> packet
> >> loss) while jumping from amazon to I2 to gigapop to 140.90.111.36 also
> >> encounters issues, but not as severe..
> >>
> >>
> >>
> >> hopefully this may help J Happy Friday J
> >>
> >>
> >>
> >> cheers,
> >>
> >>
> >>
> >> --patrick
> >>
> >>
> >>
> >> -------------------------------------------------------
> >>
> >> Patrick L. Francis
> >>
> >> Vice President of Research & Development
> >>
> >>
> >>
> >> Aeris Weather
> >>
> >>
> >>
> >> http://aerisweather.com/
> >>
> >> http://modelweather.com/
> >>
> >>
> >>
> >> http://facebook.com/wxprofessor/
> >>
> >>
> >>
> >> --------------------------------------------------------
> >>
> >>
> >>
> >> *From:* conduit-bounces@xxxxxxxxxxxxxxxx [mailto:
> >> conduit-bounces@xxxxxxxxxxxxxxxx] *On Behalf Of *Arthur A Person
> >> *Sent:* Friday, February 19, 2016 1:57 PM
> >> *To:* Pete Pokrandt <poker@xxxxxxxxxxxx>
> >> *Cc:* Bentley, Alicia M <ambentley@xxxxxxxxxx>; Michael Schmidt <
> >> mschmidt@xxxxxxxx>; support-conduit@xxxxxxxxxxxxxxxx <
> >> conduit@xxxxxxxxxxxxxxxx>; _NCEP.List.pmb-dataflow <
> >> ncep.list.pmb-dataflow@xxxxxxxx>; Daes Support <daessupport@xxxxxxxxxx>
> >> *Subject:* Re: [conduit] Large CONDUIT latencies to UW-Madison
> >> idd.aos.wisc.edu starting the last day or two.
> >>
> >>
> >>
> >> Pete,
> >>
> >>
> >>
> >> We've been struggling with latencies for months to the point where I've
> >> been feeding gfs 0p25 from NCEP and the rest from Unidata... that is, up
> >> untl Feb 10th. The afternoon of the 10th, our latencies to NCEP
> dropped to
> >> what I consider "normal", an average maximum latency of about 30
> seconds.
> >> Our networking folks and NCEP have been trying to identify what this
> >> problem was, but as far as I know, no problem has been identified or
> action
> >> taken. So, it appears it's all buried in the mysteries of the internet.
> >> I've switched data collection back to NCEP at this point, but I'm on the
> >> edge of my seat waiting to see if it reverts back to the old behavior...
> >>
> >>
> >>
> >> Art
> >>
> >>
> >> ------------------------------
> >>
> >> *From: *"Pete Pokrandt" <poker@xxxxxxxxxxxx>
> >> *To: *"Carissa Klemmer - NOAA Federal" <carissa.l.klemmer@xxxxxxxx>,
> >> "Arthur A Person" <aap1@xxxxxxx>, "_NCEP.List.pmb-dataflow" <
> >> ncep.list.pmb-dataflow@xxxxxxxx>
> >> *Cc: *"support-conduit@xxxxxxxxxxxxxxxx" <conduit@xxxxxxxxxxxxxxxx>,
> >> "Michael Schmidt" <mschmidt@xxxxxxxx>, "Bentley, Alicia M" <
> >> ambentley@xxxxxxxxxx>, "Daes Support" <daessupport@xxxxxxxxxx>
> >> *Sent: *Friday, February 19, 2016 12:20:20 PM
> >> *Subject: *Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu
> >> starting the last day or two.
> >>
> >> All,
> >>
> >>
> >>
> >> Not sure if this is on my end or somewhere upstream, but the last
> several
> >> runs my CONDUIT latencies have been getting huge to the point where we
> are
> >> losing data.
> >>
> >>
> >>
> >> I did stop my ldm the other day to add in an alternate feed for Gilbert
> at
> >> allisonhous.com, not sure if that pushed me over a bandwidth limit, or
> by
> >> reconnecting we got hooked up to a different remote ldm, or taking a
> >> different path, that shot the latencies up.
> >>
> >>
> >>
> >> Seems to be really only CONDUIT, none of our other feeds show this kind
> of
> >> latency.
> >>
> >>
> >>
> >> Still looking into things locally, but wanted make people aware. I just
> >> rebooted idd.aos.wisc.edu, will see if that helps at all.
> >>
> >>
> >>
> >> Here's an ldmping and traceroute from idd.aos.wisc.edu to
> >> conduit.ncep.noaa.gov.
> >>
> >>
> >>
> >> [ldm@idd ~]$ ldmping conduit.ncep.noaa.gov
> >>
> >> Feb 19 17:16:08 INFO: State Elapsed Port Remote_Host
> >> rpc_stat
> >>
> >> Feb 19 17:16:08 INFO: Resolving conduit.ncep.noaa.gov to 140.90.101.42
> >> took 0.00486 seconds
> >>
> >> Feb 19 17:16:08 INFO: RESPONDING 0.115499 388
> conduit.ncep.noaa.gov
> >>
> >>
> >>
> >>
> >>
> >> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
> byte
> >> packets
> >>
> >> 1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)
> >> 0.760 ms 0.954 ms 0.991 ms
> >>
> >> 2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.119 ms
> >> 18.123 ms 18.107 ms
> >>
> >> 3 et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9) 27.836 ms
> >> 27.852 ms 27.838 ms
> >>
> >> 4 et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4) 37.363 ms
> >> 37.363 ms 37.345 ms
> >>
> >> 5 noaa-i2.demarc.maxgigapop.net (206.196.177.118) 38.051 ms 38.254
> ms
> >> 38.401 ms
> >>
> >> 6 140.90.111.36 (140.90.111.36) 118.042 ms 118.412 ms 118.529 ms
> >>
> >> 7 140.90.76.69 (140.90.76.69) 41.764 ms 40.343 ms 40.500 ms
> >>
> >> 8 * * *
> >>
> >> 9 * * *
> >>
> >> 10 * * *
> >>
> >>
> >>
> >> Similarly to ncepldm
> >>
> >>
> >>
> >> [ldm@idd ~]$ ldmping ncepldm4.woc.noaa.gov
> >>
> >> Feb 19 17:18:40 INFO: State Elapsed Port Remote_Host
> >> rpc_stat
> >>
> >> Feb 19 17:18:40 INFO: Resolving ncepldm4.woc.noaa.gov to 140.172.17.205
> >> took 0.001599 seconds
> >>
> >> Feb 19 17:18:40 INFO: RESPONDING 0.088901 388
> ncepldm4.woc.noaa.gov
> >>
> >> ^C
> >>
> >>
> >>
> >> [ldm@idd ~]$ traceroute ncepldm4.woc.noaa.gov
> >>
> >> traceroute to ncepldm4.woc.noaa.gov (140.172.17.205), 30 hops max, 60
> >> byte packets
> >>
> >> 1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)
> >> 0.730 ms 0.831 ms 0.876 ms
> >>
> >> 2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.092 ms
> >> 18.092 ms 18.080 ms
> >>
> >> 3 ae0.3454.core-l3.frgp.net (192.43.217.223) 40.196 ms 40.226 ms
> >> 40.256 ms
> >>
> >> 4 noaa-i2.frgp.net (128.117.243.11) 40.970 ms 41.012 ms 40.996 ms
> >>
> >> 5 2001-mlx8-eth-1-2.boulder.noaa.gov (140.172.2.18) 42.780 ms
> 42.778
> >> ms 42.764 ms
> >>
> >> 6 mdf-rtr-6.boulder.noaa.gov (140.172.6.251) 40.869 ms 40.922 ms
> >> 40.946 ms
> >>
> >> 7 * * *
> >>
> >> 8 * * *
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Pete
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Pete Pokrandt - Systems Programmer
> >> UW-Madison Dept of Atmospheric and Oceanic Sciences
> >> 608-262-3086 - poker@xxxxxxxxxxxx
> >>
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Arthur A. Person
> >> Research Assistant, System Administrator
> >> Penn State Department of Meteorology
> >> email: aap1@xxxxxxx, phone: 814-863-1563
> >>
> >> _______________________________________________
> >> conduit mailing list
> >> conduit@xxxxxxxxxxxxxxxx
> >> For list information or to unsubscribe, visit:
> >> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >
> >
> >
> > --
> > Gerry Creager
> > NSSL/CIMMS
> > 405.325.6371
> > ++++++++++++++++++++++
> > “Big whorls have little whorls,
> > That feed on their velocity;
> > And little whorls have lesser whorls,
> > And so on to viscosity.”
> > Lewis Fry Richardson (1881-1953)
> >
>
>
> _______________________________________________
> conduit mailing list
> conduit@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
