Well, it turns out that our call manager was taking a coffee break every now and then, causing some serious hangovers for our users. They faced slow page loading and degraded call quality.

We started our Sherlock Holmes investigation. As the 1st step of the t-shoot, we took the source IP and destination IP. Asked a user to ping the destination (CUCM) from their laptop and didn’t observe any packet drops or delays.

So, we decided to put on our mining helmets and dig deeper. Lo and behold, when we logged in to the user’s location gateway and pinged the call manager, we discovered that every sixth packet was getting dropped. We were under the impression that because of this, the users faced issues. However, they had mentioned it was an intermittent issue, but these drops were periodic.

The below ping is towards CUCM Server (10.10.10.10) and Google DNS (8.8.8.8) sending 100 packets.

gw1#ping 10.10.10.10 re 100           
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!
!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!
Success rate is 84 percent (84/100), round-trip min/avg/max = 13/13/14 ms
gw1#
gw1#ping 8.8.8.8 re 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 23/23/25 ms
gw1#

Upon further discussion, we found out that this is due to ICMP rate limiting done in the CUCM firewall policies. This behaviour is normal and expected when pinged from IOS devices as they send pings at a faster rate. If the CUCM is pinged from Windows/MAC laptops there are no drops observed.

Below is the CLI output of a CUCM firewall policy

admin:utils firewall ipv4 list
***output Omitted***
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 10/sec burst 5
LOG icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 8 limit: avg 1/min burst 5 LOG flags 0 level 4 prefix `ping flood '
DROP icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 8
***output Omitted***

Courtecy: Cisco community

Although the issue happened due to an underlying call routing, there wasn’t any issue in the network carrying voice traffic.

Now, comes the question of why ICMP rate limiting is done.

ICMP traffic can cause network congestion and potentially disrupt voice traffic. For example, a flood of ICMP traffic from a source IP address can overwhelm network resources and cause delays or packet loss for voice traffic. To prevent this, CUCM firewalls are configured to limit the rate of ICMP traffic that is allowed to pass through. This can help ensure that important voice traffic is prioritized over ICMP traffic and is not impacted by excessive network congestion caused by ICMP traffic.

In addition, limiting ICMP traffic can also help prevent malicious attacks such as ICMP flooding, which is a common form of DoS attack. By controlling the rate of ICMP traffic, CUCM firewalls can reduce the impact of these attacks and help ensure the availability and reliability of the voice communications system.

It’s like a bouncer at a club deciding who gets in based on their dance moves. But hey, it’s all for a good cause - preventing network congestion and ensuring voice traffic gets top priority, like a VIP at a concert. And let’s face it, no one wants their voice traffic to sound like a robot on a bad connection due to congestion.