Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NANOG: users

ICMP Redirects from residential customer subnets?

 

 

NANOG users RSS feed   Index | Next | Previous | View Threaded


ml at kenweb

May 9, 2012, 7:19 AM

Post #1 of 4 (489 views)
Permalink
ICMP Redirects from residential customer subnets?

Last night I was troubleshooting a strange issue where Apple products
(So far just MacOS and Airports) were losing internet connectivity
sporadically.

Originally I thought it was an IPv6 transition technology causing the
problem but the customer couldn't even ping their default GW via v4.

To rule out the customer mistyping/giving us wrong information on what
they were seeing I attempted to verify IP connectivity from my DHCP
server to them. I pinged the IP they had retrieved via DHCP earlier.

What I got back were ICMP redirects interspersed with echo replies from
the customer I was pinging. The redirects were of the form:

"Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the
customer I was troubleshooting. Thinking that was very odd I setup an
ACL on the vlan serving that subnet to log ICMP redirects. What I found
was one IP x.y.z.56 sending redirects to IPs on my network as well as
several IPs outside my network. As far as I know there is no legitimate
reason for a residential PC or home gateway to send ICMP redirects.
There were also a few dozen other IPs on that subnet sending ICMP
redirects. A majority of them had 68:7f:74 (Cisco-Linksys) OUIs. There
were also some Belkins and one ASUStek OUIs.

The 68:7f:74 source MACs were dispersed amongst many customers not all
from the same customer. Which leads me to believe there is either a
bugged Linksys firmware or an exploited Linksys home gateway causing
trouble.

Has anyone ever seen something like this before?

Is there any reason to see ICMP redirects on a single homed residential
subnet? I'm considering adding ICMP redirects to my customer edge ACL
unless there is a legitimate purpose for these packets.


Thanks
-ML


rps at maine

May 9, 2012, 10:10 AM

Post #2 of 4 (464 views)
Permalink
Re: ICMP Redirects from residential customer subnets? [In reply to]

This is expected and will happen if the consumer router receives traffic
not destined for it for most consumer devices.

In the Ethernet world, it's usually the result of an active MAC falling out
of the table (e.g. disconnected) before the ARP entry on the router
expires. The default behavior is to flood the unknown packet out every
port. On a Cisco switch you would be looking at using something like UUFB
(unknown unicast flood blocking).

You might want to keep an eye on resource usage on your routers if you're
seeing this problem. Without UUFB there is a considerable uptick in ARP and
ICMP traffic caused by this behavior, usually driving up CPU.




On Wed, May 9, 2012 at 10:19 AM, ML <ml [at] kenweb> wrote:

> Last night I was troubleshooting a strange issue where Apple products (So
> far just MacOS and Airports) were losing internet connectivity sporadically.
>
> Originally I thought it was an IPv6 transition technology causing the
> problem but the customer couldn't even ping their default GW via v4.
>
> To rule out the customer mistyping/giving us wrong information on what
> they were seeing I attempted to verify IP connectivity from my DHCP server
> to them. I pinged the IP they had retrieved via DHCP earlier.
>
> What I got back were ICMP redirects interspersed with echo replies from
> the customer I was pinging. The redirects were of the form:
>
> "Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the
> customer I was troubleshooting. Thinking that was very odd I setup an ACL
> on the vlan serving that subnet to log ICMP redirects. What I found was
> one IP x.y.z.56 sending redirects to IPs on my network as well as several
> IPs outside my network. As far as I know there is no legitimate reason for
> a residential PC or home gateway to send ICMP redirects. There were also a
> few dozen other IPs on that subnet sending ICMP redirects. A majority of
> them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and
> one ASUStek OUIs.
>
> The 68:7f:74 source MACs were dispersed amongst many customers not all
> from the same customer. Which leads me to believe there is either a bugged
> Linksys firmware or an exploited Linksys home gateway causing trouble.
>
> Has anyone ever seen something like this before?
>
> Is there any reason to see ICMP redirects on a single homed residential
> subnet? I'm considering adding ICMP redirects to my customer edge ACL
> unless there is a legitimate purpose for these packets.
>
>
> Thanks
> -ML
>
>
>
>
>
>


--
Ray Soucy

Epic Communications Specialist

Phone: +1 (207) 561-3526

Networkmaine, a Unit of the University of Maine System
http://www.networkmaine.net/


bedard.phil at gmail

May 9, 2012, 11:00 AM

Post #3 of 4 (461 views)
Permalink
Re: ICMP Redirects from residential customer subnets? [In reply to]

I've seen this reported as a bug recently with Cisco/Linksys since the device is responding to frames for which it isn't the destination MAC when it should just discard them like the below case. Not all consumer gateways do this.

But absolutely agree it is the ARP/MAC age out mismatch issue that is the likely culprit.

Phil

On May 9, 2012, at 1:10 PM, Ray Soucy <rps [at] maine> wrote:

> This is expected and will happen if the consumer router receives traffic
> not destined for it for most consumer devices.
>
> In the Ethernet world, it's usually the result of an active MAC falling out
> of the table (e.g. disconnected) before the ARP entry on the router
> expires. The default behavior is to flood the unknown packet out every
> port. On a Cisco switch you would be looking at using something like UUFB
> (unknown unicast flood blocking).
>
> You might want to keep an eye on resource usage on your routers if you're
> seeing this problem. Without UUFB there is a considerable uptick in ARP and
> ICMP traffic caused by this behavior, usually driving up CPU.
>
>
>
>
> On Wed, May 9, 2012 at 10:19 AM, ML <ml [at] kenweb> wrote:
>
>> Last night I was troubleshooting a strange issue where Apple products (So
>> far just MacOS and Airports) were losing internet connectivity sporadically.
>>
>> Originally I thought it was an IPv6 transition technology causing the
>> problem but the customer couldn't even ping their default GW via v4.
>>
>> To rule out the customer mistyping/giving us wrong information on what
>> they were seeing I attempted to verify IP connectivity from my DHCP server
>> to them. I pinged the IP they had retrieved via DHCP earlier.
>>
>> What I got back were ICMP redirects interspersed with echo replies from
>> the customer I was pinging. The redirects were of the form:
>>
>> "Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the
>> customer I was troubleshooting. Thinking that was very odd I setup an ACL
>> on the vlan serving that subnet to log ICMP redirects. What I found was
>> one IP x.y.z.56 sending redirects to IPs on my network as well as several
>> IPs outside my network. As far as I know there is no legitimate reason for
>> a residential PC or home gateway to send ICMP redirects. There were also a
>> few dozen other IPs on that subnet sending ICMP redirects. A majority of
>> them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and
>> one ASUStek OUIs.
>>
>> The 68:7f:74 source MACs were dispersed amongst many customers not all
>> from the same customer. Which leads me to believe there is either a bugged
>> Linksys firmware or an exploited Linksys home gateway causing trouble.
>>
>> Has anyone ever seen something like this before?
>>
>> Is there any reason to see ICMP redirects on a single homed residential
>> subnet? I'm considering adding ICMP redirects to my customer edge ACL
>> unless there is a legitimate purpose for these packets.
>>
>>
>> Thanks
>> -ML
>>
>>
>>
>>
>>
>>
>
>
> --
> Ray Soucy
>
> Epic Communications Specialist
>
> Phone: +1 (207) 561-3526
>
> Networkmaine, a Unit of the University of Maine System
> http://www.networkmaine.net/


jason at thebaughers

May 9, 2012, 1:24 PM

Post #4 of 4 (463 views)
Permalink
Re: ICMP Redirects from residential customer subnets? [In reply to]

I've seen this with fixed wireless base radios that are bridges. We had
cases where a customer radio would drop offline, so the base radio would
drop the mac of the customer router from it's table. New ICMP packets
coming from the network would hit the wireless base radio and be flooded
out to all the other client radios. Any other customer routers that were
Linksys WRT54G and running a specific firmware version (can't remember
the version) that had previously learned the IP of the now-missing
customer router would redirect the packet back onto the network,
reducing the TTL by 1. Since the wireless base would then flood the
packet out again, we would see a storm of traffic that would
exponentially increase until the TTL hit 0, at which time it would stop.

If I remember right, I was able to cause the Linksys to stop this
behavior by enabling the multicast filtering feature, although I'm not
sure why that did it. Since I couldn't run around to customers and
enable it on each one, I ended up filtering all ICMP from the network
towards the customers.

I researched the issue at the time and never found anyone else who had
mentioned it. I didn't even bother to approach Linksys support.

Jason


On 5/9/2012 1:00 PM, Phil wrote:
> I've seen this reported as a bug recently with Cisco/Linksys since the device is responding to frames for which it isn't the destination MAC when it should just discard them like the below case. Not all consumer gateways do this.
>
> But absolutely agree it is the ARP/MAC age out mismatch issue that is the likely culprit.
>
> Phil
>
> On May 9, 2012, at 1:10 PM, Ray Soucy<rps [at] maine> wrote:
>
>> This is expected and will happen if the consumer router receives traffic
>> not destined for it for most consumer devices.
>>
>> In the Ethernet world, it's usually the result of an active MAC falling out
>> of the table (e.g. disconnected) before the ARP entry on the router
>> expires. The default behavior is to flood the unknown packet out every
>> port. On a Cisco switch you would be looking at using something like UUFB
>> (unknown unicast flood blocking).
>>
>> You might want to keep an eye on resource usage on your routers if you're
>> seeing this problem. Without UUFB there is a considerable uptick in ARP and
>> ICMP traffic caused by this behavior, usually driving up CPU.
>>
>>
>>
>>
>> On Wed, May 9, 2012 at 10:19 AM, ML<ml [at] kenweb> wrote:
>>
>>> Last night I was troubleshooting a strange issue where Apple products (So
>>> far just MacOS and Airports) were losing internet connectivity sporadically.
>>>
>>> Originally I thought it was an IPv6 transition technology causing the
>>> problem but the customer couldn't even ping their default GW via v4.
>>>
>>> To rule out the customer mistyping/giving us wrong information on what
>>> they were seeing I attempted to verify IP connectivity from my DHCP server
>>> to them. I pinged the IP they had retrieved via DHCP earlier.
>>>
>>> What I got back were ICMP redirects interspersed with echo replies from
>>> the customer I was pinging. The redirects were of the form:
>>>
>>> "Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the
>>> customer I was troubleshooting. Thinking that was very odd I setup an ACL
>>> on the vlan serving that subnet to log ICMP redirects. What I found was
>>> one IP x.y.z.56 sending redirects to IPs on my network as well as several
>>> IPs outside my network. As far as I know there is no legitimate reason for
>>> a residential PC or home gateway to send ICMP redirects. There were also a
>>> few dozen other IPs on that subnet sending ICMP redirects. A majority of
>>> them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and
>>> one ASUStek OUIs.
>>>
>>> The 68:7f:74 source MACs were dispersed amongst many customers not all
>>> from the same customer. Which leads me to believe there is either a bugged
>>> Linksys firmware or an exploited Linksys home gateway causing trouble.
>>>
>>> Has anyone ever seen something like this before?
>>>
>>> Is there any reason to see ICMP redirects on a single homed residential
>>> subnet? I'm considering adding ICMP redirects to my customer edge ACL
>>> unless there is a legitimate purpose for these packets.
>>>
>>>
>>> Thanks
>>> -ML
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Ray Soucy
>>
>> Epic Communications Specialist
>>
>> Phone: +1 (207) 561-3526
>>
>> Networkmaine, a Unit of the University of Maine System
>> http://www.networkmaine.net/
>
>
>

NANOG users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.