Upgrading from 8.x/9.x to 10.x

Security Gateway Articles and How to's
Locked
Tomas
Posts: 34
Joined: 15 Sep 2008, 15:57
Location: Clavister HQ - Örnsköldsvik

Upgrading from 8.x/9.x to 10.x

Post by Tomas » 30 Jan 2013, 14:18

This How-to applies to:
  • Clavister CorePlus 8.x and 9.x.
  • Clavister cOS Core 10.x
Table of contents
  • Problem description
  • Highbuffers
  • ConnTimeouts
  • LogSendPerSecLimit
  • Ringsize
  • MaxConnections
  • TCP Sequence Numbers
Problem description
When upgrading the CorePlus from an older version (prior to 10.x), the old settings in the Advanced Settings section is no longer compatible with the new CorePlus. This can give a lot of different errors that might be hard to pin-point.
Some configurations have been around a long time and been upgraded to various major versions, such as 8.90, 9.x and is now facing an upgrade or the upgrade is already completed to 10.x.

It is always important to know that your current Advanced Settings are matching the version of CorePlus/cOS Core you are using.

This article will assume you have already upgraded to 10.x and we will verify the values of some settings.

Highbuffers
This setting can be found in System > Advanced Settings > Misc. Settings > Highbuffers.

The Highbuffers setting controls how much RAM is given to the system to handle connections, NIC ring buffers and a lot of other things.If this value is set too low, performance will be degraded, the buffers might be flooded (causing a restart with the message "Buffers flooded for more than 3600 seconds"), HA sync might have problems (causing log entries in the HA category stating that they can't decide who is active and who is not etc.) and a lot of other strange issues.

The HA sync problem might look like this, but it can also be caused by a bad sync cable:
  • Notice;HA;HASync connection to peer Security Gateway established.;
    Notice;HA;HASync connection to peer Security Gateway failed. Reconnecting...;
    Notice;HA;HASync connection to peer Security Gateway established.;
    Notice;HA;Conflict: both peers are inactive! Resolving...;
    Notice;HA;Both inactive, peer has fewer connections; going active...;
    Notice;HA;Peer Security Gateway disappeared.;
    Notice;HA;HASync connection to peer Security Gateway failed. Reconnecting...;
    Notice;HA;HASync connection to peer Security Gateway established.;
    Notice;HA;Peer Security Gateway is alive;
    Notice;HA;HASync connection to peer Security Gateway established.;
    Notice;HA;Conflict: both peers are inactive! Resolving...;
    Notice;HA;Both inactive, peer has more connections; staying inactive...;
    Notice;HA;Peer Security Gateway disappeared. Going active.;
    Notice;HA;Peer Security Gateway is alive;
On older systems a value of 1024 was default. In addition to this value, about 200 extra buffers was added. On larger systems it could be around 3100. This would give the following output in Remote Console:
  • > stat
    
    Uptime : 0 days, 00:05:00
    Last shutdown  : 2007-03-30 09:00:00: Buffers flooded for more
     than 3600 seconds
    CPU Load   : 0%
    Connections: 4 out of 16000
    Fragments  : 0 out of 1024 (0 lingering)
    Buffers allocated  : 3343
    Buffers memory : 3343 x 2564 = 8370 KB
    Fragbufs allocated : 32
    Fragbufs memory: 32 x 10040 = 313 KB
    Out-of-buffers : 0

Under "Buffers memory" you see 3343 (the buffers) times their size and the sum of RAM used (8 MB in this case).

Starting from CorePlus 8.70, more highbuffers are needed as more features was introduced.

The small size appliances (IXP platform based such as SG50, SG60 and Eagle 7), the maximum value you can get is around 3000-4000, no matter how high you set it statically, but these models usually have no problem using the "dynamic" setting.
For the medium sized appliances (SG4200, Wolf 3 series) a value of around 25000 is usually suitable, depending on the features used and number of connections available etc.
For the larger appliances (SG4300/4500 and Wolf 5 series) a value of around 40000 is usually suitable, depending on the features used and number of connections available etc.

The "dynamic" setting is dynamic in the sense that it is set at boot time, calculated from a number of parameters such as number of interfaces, connections etc. Experience has shown that the calculated value often is lower than the recommended values above.

If you use the dynamic setting, please verify the value with the "stat" command at the row "Buffers memory". If it is lower than you want, please disable "dynamic" and set it statically instead.

<i>Always make sure you have enough RAM available before you change this setting, as you will reserve around 50-100 MB RAM extra when you increase this setting.</i>

Changes to the HighBuffers value requires a complete reboot of the system to be applied.

ConnTimeouts
The old value here can be 3 600 or around 86 400. The new default setting is 262 144 (seconds).

LogSendPerSecLimit
The old value here is around 50. The new default setting is 2000 (log entries sent per second).

Having a too low value here on a busy Clavister, gives log messages about "Log truncated", hence you are missing vital log information.

Ringsize
This setting is especially important for the gigabit NICs in your system. In CorePlus 9.20.00 and newer, the ring settings can be applied to many different NIC types. In the example below we use the Intel E1000 and E100 as an example.

Beware! You MUST have set a rather high and static value for High buffers, and deployed it AND restarted the device, before you change these values, or you might get into RAM/High buffers related problems that will even prevent the system from booting!

The default settings are:
  • e1000_rx = 64
    e1000_tx = 256
    e100_rx  = 32
    e100_tx  = 128
If you have the RAM available (check with the "mem" command) and High buffers (check with the "stat" command), it is advisable to increase the size of the e1000 rings, but it can also be good to increase the e100 rings too:
  • e1000_rx = 512
    e1000_tx = 1024
    e100_rx  = 64
    e100_tx  = 256
Remember that having larger rings will increase the latency, because it takes longer time to find the information in a larger buffer, but it will on the other hand prevent packet loss due to full ring buffers.

As stated above, you must have a high, fixed, value on your High Buffers before you apply a setting like this.

MaxConnections
This value should be set to "dynamic" to match the capacity in the license used. However it may also be a static value. Please verify that you are aware of its current setting, and that it matches the load of the traffic.

TCP Sequence Numbers

This setting can be found in System > Advanced Settings > TCP Settings > TCP Sequence Numbers.

The TCP Sequence Numbers will make sure that the sequence numbers in a TCP connection behaves according to the TCP specification. However, many programmers out there are not obeying these specifications and having this feature enabled might cause their connections to be dropped as the sequence numbers are not matching the expected value.

Setting this value to Ignore will cause the Clavister to not stop these connections.

Locked