Re: [ldm-users] GOES-16 reception issues when going through an enterprise level switch

  • To: "Gilbert Sebenste" <gilbert@xxxxxxx>
  • Subject: Re: [ldm-users] GOES-16 reception issues when going through an enterprise level switch
  • From: admin@xxxxxxxx
  • Date: Tue, 14 Mar 2017 16:54:58 -0400 (EDT)
This is common in enterprise switches for that filtering to be ON by default, 
after some malware appeared that used multicast packets to do denial of service.

Ray Weber

On Tuesday, March 14, 2017 4:50pm, "Gilbert Sebenste" <gilbert@xxxxxxx> said:

> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 
> 
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/ Hey Gerry,
> 
> It’s an Avaya 5510 (not 5520, sorry---the one has half of the number of
> ports, but is otherwise the same as the 20).
> By default, it rejects the flood of packets on those encapsulated IP’s. And
> I now realize, COD isn’t the only one having this issue…
> 
> Gilbert
> 
> Gilbert Sebenste
> Staff Meteorologist
> 
> Environmental Health and Safety
> Labs for Wellness 154 | DeKalb, Illinois 60115
> 815-753-5492
> gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>
> http://weather.admin.niu.edu<http://weather.admin.niu.edu/>
> 
> Everyone. Home. Safely.
> 
> [NIU]
> 
> 
> From: Gerry Creager - NOAA Affiliate [mailto:gerry.creager@xxxxxxxx]
> Sent: Tuesday, March 14, 2017 3:38 PM
> To: Gilbert Sebenste <gilbert@xxxxxxx>
> Cc: ldm-users@xxxxxxxxxxxxxxxx; NOAAPORT <noaaport@xxxxxxxxxxxxxxxx>
> Subject: Re: [ldm-users] GOES-16 reception issues when going through an 
> enterprise
> level switch
> 
> Who'd have thought the switch wouldn't pass multicast? Is it possible the 
> campus
> network geeks told it not to? Using multicast for disseminating data like that
> "just makes sense".
> Glad you found THAT problem. What brand of switch, anyway? Cisco?
> gerry
> 
> On Tue, Mar 14, 2017 at 3:30 PM, Gilbert Sebenste
> <gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>> wrote:
> Hello everyone,
> 
> The College of DuPage has a NOAAport receiver with a Novra S300N, and two 
> Ubuntu
> boxes as ingestors, so that if one fails, the other will still provide data. 
> To
> accomplish this, they use an enterprise level network switch, a Nortel/Avaya 
> 5520,
> that costs a lot of money (and not the $50 gigabit switches you can get from
> Newegg, or wherever), to split the data feed so that both servers can get and
> ultimately relay the data.
> 
> When trying to receive the test GOES-16 imagery months ago, it wasn’t coming
> across their NOAAport feed. I thought it was a glitch with their Novra box, 
> or the
> network switch. But now that GOES-16 data is being sent, neither NOAAPort 
> ingest
> server were getting the GOES-16 data on the NOTHER feed. What could be wrong?
> 
> We restarted the LDM on both servers, deleting and remaking the queues. That
> didn’t do anything.
> We then double checked the ldmd.conf files, which had a few minor issues, and 
> also
> checked the logs,
> of course. Still, no GOES-16 data was showing up.
> We then did a packet sniff from one of the servers to the switch. No GOES-16 
> test
> data was being sent to the boxes from the switch!
> 
> We then made sure the Ubuntu kernel parameters were set correctly, per the LDM
> instructions. One box didn’t have it correct; the other was correct. We
> rebooted both boxes after doing checks. That didn’t get us GOES-16 data,
> either.
> 
> So then, today, I convinced a College of DuPage meteorology program employee 
> to go
> out to the bitter cold dish farm and climate-controlled “dog house”
> where the Novra and the servers are. As he power-cycled the Novra, I was 
> doing an
> ldmadmin watch on the NOTHER feed on the ingest servers. When the Novra came 
> back
> up after the power cycke…no GOES-16 data was coming across to the servers!
> 
> So then we looked to the network switch. The employee grabbed a jumper cable, 
> and
> ran an Ethernet cable directly from the Novra to one of the two ingest 
> servers.
> Then, finally, GOES-16 data started pouring in on that server!
> 
> But why? How can a $1,000+ enterprise level switch have a problem like this?
> 
> Well, as it turns out, it wasn’t the fault of the switch at all. See:
> 224.0.1.[1-10] have reserved internet protocols on them:
> http://www.networksorcery.com/enp/protocol/ip/multicast.htm
> 
> For those of you who don’t like to understandably click on links from
> strangers (and geeky meteorologists):
> 
>                                                                 INTERNETWORK
> CONTROL BLOCK
> 
> 224.0.1.0
> 
> VMTP<http://www.networksorcery.com/enp/protocol/vmtp.htm> Managers.
> 
> RFC 1045
> 
> 224.0.1.1
> 
> NTP<http://www.networksorcery.com/enp/protocol/ntp.htm>, Network Time 
> Protocol.
> 
> RFC 1119<http://www.networksorcery.com/enp/rfc/rfc1119.pdf>
> 
> 224.0.1.2
> 
> SGI-Dogfight.
> 
> 
> 
> 224.0.1.3
> 
> Rwhod.
> 
> 
> 
> 224.0.1.4
> 
> VNP.
> 
> 
> 
> 224.0.1.5
> 
> Artificial Horizons - Aviator.
> 
> 
> 
> 224.0.1.6
> 
> NSS, Name Service Server.
> 
> 
> 
> 224.0.1.7
> 
> AUDIONEWS - Audio News Multicast.
> 
> 
> 
> 224.0.1.8
> 
> SUN NIS+ Information Service.
> 
> 
> 
> 224.0.1.9
> 
> MTP<http://www.networksorcery.com/enp/protocol/mtp.htm>, Multicast Transport
> Protocol.
> 
> RFC 1301<http://www.networksorcery.com/enp/rfc/rfc1301.txt>
> 
> 224.0.1.10
> 
> IETF-1-LOW-AUDIO.
> 
> 
> Uh huh. When the NWS picked those IP addresses to encapsulate the data, they
> picked IP numbers that are used and reserved for enterprise-level switches.
> 
> As for the College of DuPage? I just got this email from head network 
> engineer,
> Dave Bukowski, who came up with this solution: you need to enable unknown
> multicast flooding on those IP’s. Here’s what Dave did, and his email
> with the solution:
> -----------
> 
> FOUND IT!!!!
> 
> 
> This was in the config
> 
> show running-config
> .....SNIP.....
> !
> ! *** IGMP ***
> !
> vlan igmp unknown-mcast-allow-flood 224.0.1.1
> vlan igmp unknown-mcast-allow-flood 224.0.1.2
> vlan igmp unknown-mcast-allow-flood 224.0.1.3
> vlan igmp unknown-mcast-allow-flood 224.0.1.4
> vlan igmp unknown-mcast-allow-flood 224.0.1.5
> vlan igmp unknown-mcast-no-flood enable
> .....ENDSNIP.....
> 
> 
> Added
> vlan igmp unknown-mcast-allow-flood 224.0.1.6
> vlan igmp unknown-mcast-allow-flood 224.0.1.7
> vlan igmp unknown-mcast-allow-flood 224.0.1.8
> vlan igmp unknown-mcast-allow-flood 224.0.1.9
> vlan igmp unknown-mcast-allow-flood 224.0.1.10
> vlan igmp unknown-mcast-allow-flood 224.0.1.201
> 
> 
> 
> So now:
> 
> SDBDF01#show vlan igmp unknown-mcast-allow-flood
> Allowed Multicast Addresses
> -----------------------------------------------------------------------
> 224.0.1.1       224.0.1.2       224.0.1.3       224.0.1.4
> 224.0.1.5       224.0.1.6       224.0.1.7       224.0.1.8
> 224.0.1.9       224.0.1.10     224.0.1.201
> Total Multicast MAC Addresses: 0
> Total Multicast IP  Addresses: 11
> 
> 
> Now we have a flood of data coming in on both noaaport1 and noaaport2. It was
> multicast and we were blocking the flood, but had to specifically allow the
> flooding by multicast address.
> 
> -Dave
> 
> 
> Thanks, Dave. We spent at least 10 hours trying to figure this out. On a $50
> switch, you might not have a problem, but if you do decide to have multiple
> ingesters by going through a network switch…if you don’t get all of
> the data, your reception may be fine…it’s the switch that’s
> been programmed to stop the flow of data.
> 
> Thanks to Tom Yoksas as he helped us try to figure this mess out, to Mike 
> Zuranski
> for going out in the bitter cold and spending hours on the phone as we tried 
> to
> figure this out, and for Dave Bukowski who ran the ball for the touchdown 
> after we
> discovered it was the switch blocking the data.
> 
> Gilbert
> Gilbert Sebenste
> Staff Meteorologist
> 
> Environmental Health and Safety
> Labs for Wellness 154 | DeKalb, Illinois 60115
> 815-753-5492<tel:(815)%20753-5492>
> gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>
> http://weather.admin.niu.edu<http://weather.admin.niu.edu/>
> Everyone. Home. Safely.
> 
> [NIU]
> 
> 
> 
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 
> 
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> 
> 
> 
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
> 




  • 2017 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: