How can I display Current bandwidth usage per IP?

Security Gateway Discussions
Post Reply
SECOIT GmbH
Posts: 32
Joined: 13 Feb 2018, 16:20
Contact:

How can I display Current bandwidth usage per IP?

Post by SECOIT GmbH » 22 Jun 2018, 15:04

Hi All,

If you are an admin I'm nearly 100% sure you heard this question already from all your customers: "Why is the internet so slow"?

Normally with all the other firewalls I have been working with in the past there is a more or less fancy GUI element that displays current connections together with their current bandwidth usage and in most cases you can sort by KB/s or similar.

How can I get the same with cOS Core? I'm sure that also cOS Core admins get this questions from their customers (as I did yesterday first time from one of our customers who have a Clavister GW).

- There is the connections tab on the GUI and the connections command. Nice - but doesn't display current bandwith usage.

- I also tried setting up two pipes, one for in, one for out, both with no limit and I assigned two pipe rules to them to capture all www traffic in both directions. Using the pipe command I can now see the bandwidth usage - but not the connections who actually consume the bandwidth.

- I can check the logs and cubes in InControl but they only give me a picture after the connection has closed and nothing real-time.


So... since I'm nearly sure that everyone of you had this internet performance question at some point in time already... How do you get this information out of cOS Core?

Something like
Source:10.3.4.5:6654 Dest:55.66.77.88:443 Prot:TCP Bandwidth:56kB/s

Ideally ordered by bandwidth.
Fancy GUI would be nice but CLI would do, too.


Thanks,
Michael
--
Michael Steffens
SECOIT GmbH
https://www.secoit.de

anders s
Posts: 33
Joined: 27 Sep 2011, 14:41

Re: How can I display Current bandwidth usage per IP?

Post by anders s » 23 Jul 2018, 11:06

There is no good solution that I am aware of. You can create a monitoring dashboard that shows which interface/vlan is using the bandwidth, start a packet capture on that interface and analyze in Wireshark to see which connection is using the most bandwidth.

Peter
Posts: 668
Joined: 10 Apr 2008, 14:14
Location: Clavister HQ - Örnsköldsvik

Re: How can I display Current bandwidth usage per IP?

Post by Peter » 16 Aug 2018, 16:12

Hi.

As Anders mentioned there is no good/optimal way to get this information. In cOS Core version 12.00.10 we do however have two new subcommands to the "connections" command called "datamore" and "dataless". These subcommands can be used to list data that has been sent in existing connections. It's not real-time but it may give you some clues as to which machines that are talking the most.
Firewall:/> connections -show -datamore=10m
State    Proto   Source                      Destination                 Tmout  Bytes   Application
-------- ------- --------------------------- --------------------------- ------ ------- -----------------
RAWIP    ESP     core:25.150.238.154:0        WAN:82.131.42.101:0            130    195M unknown
RAWIP    ESP     WAN:221.89.54.169:0         core:25.150.238.154:0           130   1.25G unknown
UDP      UDP     core:192.168.4.254:999      To_Lab:192.168.98.50:998       128   15.1M unknown
TCP_OPEN TCP     VLAN10:192.168.4.200:49826  WAN:114.199.65.199:4070     262144   30.7M spotify
TCP_OPEN TCP     VLAN10:192.168.4.158:51081  WAN:114.199.64.50:443       246746   21.4M unknown
UDP      UDP     to_tech_home:192.168.1.1:999 To_Lab:192.168.98.50:998       130    101M unknown 
This command can then also be used in combination with other filters such as source interface and whatnot to narrow down the searches even more.

But it is still not real-time, real-time data means keeping track of all connections at all times and that can cause big performance hits.

In Stream we have a flow command that can give you pretty much real-time data in an existing flow (connection), it can look something like this:
System:/> flow -show -usage
         Source                      Dest                   ID /
Proto    Iface   IP           Port   Iface  IP              Port  Timeout  Pkst  Bytes
-------  ------  -----------  -----  -----  --------------  ----  -------  ----  -----
TCP      mgmt    172.16.5.40  49361  core   172.16.100.178  22     262140  880   105k

System:/> flow -show -usage -verbose
         Source                      Dest                     ID /
Proto    Iface   IP           Port   Iface    IP              Port   Timeout  Flags  Pkst  Bytes
-------  ------  -----------  -----  -------  --------------  -----  -------  -----  ----  -----
TCP      mgmt    172.16.5.40  49361  core     172.16.100.178  22      262141  Pp     902   108k
...rev:  mgmt    172.16.5.40  49361  core(0)  127.0.0.1       62001   262141  Pp     363   62.3k
But in cOS Core we do not have the above functionality.

Also as you mentioned is that this is not an unusual question, trying to find a spammer or infected PC in a large network can be quite tricky. Using Dashboards, Analyzer cube, logs may not be enough to narrow it down enough. What i have done in several of these situations is to take a pcap sample and then analyze that sample in Wireshark.

Lets say you have determined that the interface that is receiving the large amount of data is the interface called "Lan". So we take a small sample (10 MByte) from the Lan interface using the WebUI or the CLI:

"pcapdump -start lan -size=10000"

Such a sample will most likely get full within miliseconds but it will still provide us with a sample of the traffic pattern for us to analyze, an alternative would be to use a subcommand called "-snaplen" to limit the amount of data captured in each packet.

"pcapdump -start lan -size=10000 -snaplen=100"

The above means we only capture the 100 first bytes in the packet. It can be useful to get a higher PPS count instead of getting all the data payload in each packet and thus causing the PCAP buffer to be full that much quicker.

Once you got the sample, load it in Wireshark then go to "Statistics->Conversation list->IPv4". There you will get quite a nice overview of all the IP's and how much they have conversed with each other. By filtering on e.g. Bytes or Packets you can get a nice summary of who is doing what in the network. You may want to take a few samples though to make sure that what you are seeing is not just a fluke or hickup in the network that happened just at the time. With this data you should hopefully be able to determine who the machine(s) that is generating the most amount of packets or bandwith in the network.

Best regards
/Peter

SECOIT GmbH
Posts: 32
Joined: 13 Feb 2018, 16:20
Contact:

Re: How can I display Current bandwidth usage per IP?

Post by SECOIT GmbH » 21 Aug 2018, 11:40

Hi Peter,

Many thanks for the hint!
What you described with PCAP is a workaround. Not as nice as the competition would do but I can work with it and to be honest - as simple as your idea is - I didn't think of it. So many thanks again!

Is there also something similar that could be done with bigger environments where you would usually use LACP? Since packet capture on the LACP interface with cOS Core doesn't work would there be another option to achieve something similar?

But anyway, your idea also makes me think that analyzing a small amount of captured data (it's limited to only 512 MByte anyway) could be done within the Web GUI. So when you are saying to constantly track connections is too ressource intensive (although this feature is included in most professional firewalls) I understand Clavisters architecture and system ressources differ from other brands. But what you suggested with the PCAP could be a feature that could be implemented within the GUI with only limited ressources I believe.
Something like a menu option to list top talkers where you can trace data for a few seconds (max 5 seconds or so) and then have the statistics calculated by cOS Core (instead of Wireshark).

Do you think it is worth putting that in as feature request?


Thanks,
Michael
--
Michael Steffens
SECOIT GmbH
https://www.secoit.de

Peter
Posts: 668
Joined: 10 Apr 2008, 14:14
Location: Clavister HQ - Örnsköldsvik

Re: How can I display Current bandwidth usage per IP?

Post by Peter » 22 Aug 2018, 14:07

Hello.

There exist a feature request (RFE) about this particular functionality (COP-14710). I have added your thoughts / feedback in the developer case.

An idea in this RFE would be to implement a sort of sample functionality. Even if it may cause performance or network hickups it could be something that can be gathered temporary. For instance sample the network on one or more interfaces during a fixed amount of time and present the data. Even if it could cause a small network interruption it is something the administrator does and have control over when it is performed. So it's nothing unexpected and can be planned if needed.

The same idea can also be applied to measuring CPU, but that is a different RFE (COP-15145).

Best regards
/Peter

SECOIT GmbH
Posts: 32
Joined: 13 Feb 2018, 16:20
Contact:

Re: How can I display Current bandwidth usage per IP?

Post by SECOIT GmbH » 22 Aug 2018, 15:45

Hi Peter,

Many thanks for letting me know! Good to know it's being processed.

For that other part of my question that I wrote above: One of our bigger customers for example where I use HA and LACP (so redundant firewalls connected to stacked/redundant switches) is located pretty much in the middle of nowhere and for the time being they have a tiny www line (12/1 MBit/s) and I keep getting the questions to investigate top talkers.

With LACP the packet capture in cOS core doesn't work (well, it shows more or less random data from the physical interfaces which is kind of "doesn't work").
What options do I have with bigger customers without any possibility to capture data?

(two more projects are already planned with HA and LACP and the devices are already purchased so this will come up even more often in the near future)


Thanks,
Michael
--
Michael Steffens
SECOIT GmbH
https://www.secoit.de

Peter
Posts: 668
Joined: 10 Apr 2008, 14:14
Location: Clavister HQ - Örnsköldsvik

Re: How can I display Current bandwidth usage per IP?

Post by Peter » 23 Aug 2018, 16:12

Hello.

When it comes to top talkers i would say that your best bet there would be to use InControl and the Analyzer datacube. Even though it is not real-time data, the datacube can provide you with top-talkers based on the data available in the logs.

Top-talkers is actually one of the primary reasons we made the datacube in the first place. It is data that many users want to have/analyse and if we are dealing with terrabytes of data, the logs is the most viable way to compile and present this data in a good way.

Troubleshooting in real-time then becomes more tricky as we cannot use the PCAP method (and maybe the amount of data is so large that it is not viable anyway), then i'm afraid i do not have any good suggestions other than maybe check it the CLI command for "datamore/dataless" could be useful.

Best regards
/Peter

zanespero
Posts: 1
Joined: 29 Nov 2018, 12:48
Contact:

Re: How can I display Current bandwidth usage per IP?

Post by zanespero » 29 Nov 2018, 13:57

Hi everybody. So, over the past few months, my family's monthly data usage limit has been grossly high. We pay for 250 gigs a month and lately have been having usages of over 320 gigs. I just need to know how to monitor web traffic on an ip address basis so I can pick out the culprit. I have already looked into dd-wrt scripts and alternate router firmwares. The dd-wrt scripts ended in failure and I couldn't seem to figure out the alternate router firmwares (such as gargoyle). Is there anything I can do? I would rather leave my main router alone for the most part. I have 3 other crappy routers that are not doing anything that I have at my disposal. Any ideas? Thanks!

Post Reply