We upgraded one of our core switch from a 4507R to a 6509-E. While moving a 3750 distribution stack switch connection, we changed the uplink port of the stack from 1/0/25 to 4/0/25 (from switch 1 to switch 4). Switch 2 is the stack master switch.
The strangest thing happened: process CPU shot up to ~90% compare with ~10% as before.
3750_Stack#sho proc cpu | exc 0.00
CPU utilization for five seconds: 88%/15%; one minute: 62%; five minutes: 62%PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
51 11511131023271962 1 0.31% 0.19% 0.24% 0 Fifo Error Detec
81 363750 181272000 2 0.15% 0.03% 0.04% 0 hpm main process
85 4191815 30304200 138 0.31% 0.13% 0.10% 0 hpm counter proc
86 16179917 86606818 186 0.15% 0.08% 0.06% 0 HRPC pm-counters
116 2304082 454784198 5 6.38% 1.71% 1.52% 0 Hulc LED Process
169 17070389 78940409 216 60.54% 42.65% 42.44% 0 IP Input
Where IP Input process CPU Utilization is way off the norm.
Reading up on troubleshooting high CPU on Cisco routers, troubleshooting High CPU Utilization in IP Input Process, troubleshooting high CPU on Catalyst 3750 Series Switches, and supportwiki 3750 high CPU didn't yield any fruitful result.
Further searching on bug tool kit resulted in a bug that has similar description to what we ran into:"Catalyst3750 removes its hardware forwarding information in certain condition. This will result in software forwarding" which explains why our CPU is so high. The suggested work around is to shutdown and bring back up the interface in question. We did that and boom! the process CPU come back down to normal again!
I learned a thing or two about Catalyst 3750 Series Stackwise operation:
Troubleshooting with Stack, and commands such as "show platform..." and "remote command".
No comments:
Post a Comment