We recently added 10G ports to our W30 and also moved some extra routing to the firewall. We now see an average CPU utilization around 60-70%. So, my question is are there any recommendation in regard of CPU utilization. What is a reasonable level and are such levels (avg 60%) prone to add delay in the network?
We do have some performance issues but they could also be related to a legacy switching environment so I’m trying to pinpoint bottlenecks.
Thanks,
Jonas
Recommended average CPU utilization.
Re: Recommended average CPU utilization.
Hello.
A tricky question, one that could have many answers as the amount of variables here can be quite big.
But....
A semi-good way for you to check if one or more interfaces is "overloaded" is to look at what the hardware driver reports back regarding incoming packets. If the driver itself report back that it is missing packets, it is an indication that the interface is overloaded and is unable to process all the packets that arrive on it. Which could lead to packet losses.
An example from a VSG:
2. If the CPU is not at 100% and you see missed packets on the hardware statistics it is an indication that there was a big packet burst were not all packets was able to be processed. This is not unusual for interfaces under heavy load, the packets missed counter will most likely increase on and off. As long as the number of missed packets is not high and is not constantly increasing, it's quite normal.
3. If the CPU is 100% and there is no missed packets on the hardware statistics it is an indication that the packet loss problem "may" be related to the Firewall. The Firewall seems to be able to handle all incoming packets but any Firewall running at 100% CPU is a problem and that needs to be investigated, it is probably not IF but WHEN you will start to have problems during a 100% state.
4. And of course, if 100% CPU and lots of missed packets it is a very strong indicator that the system is overloaded and packet losses are pretty much guaranteed on one or several interfaces.
Note: The hardware statistics are different based on driver, for instance on Realtek driver it is called "Missed Frames".
Best regards
/Peter
A tricky question, one that could have many answers as the amount of variables here can be quite big.
But....
A semi-good way for you to check if one or more interfaces is "overloaded" is to look at what the hardware driver reports back regarding incoming packets. If the driver itself report back that it is missing packets, it is an indication that the interface is overloaded and is unable to process all the packets that arrive on it. Which could lead to packet losses.
An example from a VSG:
Iface If1 - Autogenerated: "E1000" (PCI Port:0 Slot:17 Bus:0) Builtin e1000 - 82545EM Gigabit Ethernet Controller (Copper) Bus 0 Slot 17 Port 0 IRQ 9 Media : "Autonegotiate" Link Status : 1000 Mbps Full duplex Receive Mode : Normal MTU : 1500 IPv4 Address : 192.168.95.14 MAC Address : 00-0C-29-2F-10-0B PBR Membership: main Software Statistics: Soft received : 653113 Soft sent : 205693 Send failures : 0 Dropped : 226 IP Input Errs : 0 Driver information / hardware statistics: IN : packets= 618653 bytes=90710754 errors= 0 dropped= 0 OUT: packets= 194981 bytes=18046963 errors= 0 dropped= 0 Collisions : 0 In : Length Errors : 0 In : Overruns : 0 In : CRC Errors : 0 In : Frame Errors : 0 In : FIFO Overruns : 0 In : Packets Missed : 0 <--- Look at this one Out: Sends Aborted : 0 Out: Carrier Errors : 0 Out: FIFO Underruns : 0 Out: Late Collisions : 0 Not using polling1. If the CPU is not 100% and you have large amounts of missed packets on the hardware statistics that is constantly increasing it is an indication of an individual interface being overloaded. This would be solved by features/functions such as Ling Aggregation / interface teaming.
2. If the CPU is not at 100% and you see missed packets on the hardware statistics it is an indication that there was a big packet burst were not all packets was able to be processed. This is not unusual for interfaces under heavy load, the packets missed counter will most likely increase on and off. As long as the number of missed packets is not high and is not constantly increasing, it's quite normal.
3. If the CPU is 100% and there is no missed packets on the hardware statistics it is an indication that the packet loss problem "may" be related to the Firewall. The Firewall seems to be able to handle all incoming packets but any Firewall running at 100% CPU is a problem and that needs to be investigated, it is probably not IF but WHEN you will start to have problems during a 100% state.
4. And of course, if 100% CPU and lots of missed packets it is a very strong indicator that the system is overloaded and packet losses are pretty much guaranteed on one or several interfaces.
Note: The hardware statistics are different based on driver, for instance on Realtek driver it is called "Missed Frames".
Best regards
/Peter
Re: Recommended average CPU utilization.
What command did you use to get that information Peter?
Re: Recommended average CPU utilization.
Ah, sorry. It was the "ifstat <iface>" CLI command.
/Peter
/Peter