Recommended average CPU utilization.

[jonasT]

We recently added 10G ports to our W30 and also moved some extra routing to the firewall. We now see an average CPU utilization around 60-70%. So, my question is are there any recommendation in regard of CPU utilization. What is a reasonable level and are such levels (avg 60%) prone to add delay in the network?

We do have some performance issues but they could also be related to a legacy switching environment so I’m trying to pinpoint bottlenecks.

Thanks,
Jonas

[Peter]

Hello.

A tricky question, one that could have many answers as the amount of variables here can be quite big.

But....

A semi-good way for you to check if one or more interfaces is "overloaded" is to look at what the hardware driver reports back regarding incoming packets. If the driver itself report back that it is missing packets, it is an indication that the interface is overloaded and is unable to process all the packets that arrive on it. Which could lead to packet losses.

An example from a VSG:

Iface If1 - Autogenerated: "E1000"  (PCI Port:0 Slot:17 Bus:0)
  Builtin e1000 - 82545EM Gigabit Ethernet Controller (Copper)  Bus 0 Slot 17 Port 0 IRQ 9
  Media           : "Autonegotiate"
  Link Status     : 1000 Mbps Full duplex
  Receive Mode    : Normal
  MTU             : 1500
  IPv4 Address    : 192.168.95.14
  MAC Address     : 00-0C-29-2F-10-0B
  PBR Membership: main

Software Statistics:
  Soft received :  653113  Soft sent     :  205693  Send failures :       0
  Dropped       :     226  IP Input Errs :       0

Driver information / hardware statistics:
  IN : packets=  618653   bytes=90710754   errors=       0   dropped=       0
  OUT: packets=  194981   bytes=18046963   errors=       0   dropped=       0
  Collisions            :        0
  In : Length Errors    :        0
  In : Overruns         :        0
  In : CRC Errors       :        0
  In : Frame Errors     :        0
  In : FIFO Overruns    :        0
  In : Packets Missed   :        0 <--- Look at this one
  Out: Sends Aborted    :        0
  Out: Carrier Errors   :        0
  Out: FIFO Underruns   :        0
  Out: Late Collisions  :        0
  Not using polling

1. If the CPU is not 100% and you have large amounts of missed packets on the hardware statistics that is constantly increasing it is an indication of an individual interface being overloaded. This would be solved by features/functions such as Ling Aggregation / interface teaming.
2. If the CPU is not at 100% and you see missed packets on the hardware statistics it is an indication that there was a big packet burst were not all packets was able to be processed. This is not unusual for interfaces under heavy load, the packets missed counter will most likely increase on and off. As long as the number of missed packets is not high and is not constantly increasing, it's quite normal.
3. If the CPU is 100% and there is no missed packets on the hardware statistics it is an indication that the packet loss problem "may" be related to the Firewall. The Firewall seems to be able to handle all incoming packets but any Firewall running at 100% CPU is a problem and that needs to be investigated, it is probably not IF but WHEN you will start to have problems during a 100% state.
4. And of course, if 100% CPU and lots of missed packets it is a very strong indicator that the system is overloaded and packet losses are pretty much guaranteed on one or several interfaces.

Note: The hardware statistics are different based on driver, for instance on Realtek driver it is called "Missed Frames".

Best regards
/Peter

[Bschoeck]

What command did you use to get that information Peter?

[Peter]

Ah, sorry. It was the "ifstat <iface>" CLI command.

/Peter