[RADIATOR] Handling surge in connections on proxy radius

[Heikki Vatiainen]

On 02/12/2014 11:32 AM, Ryan wrote:

> On the proxy radius, running 'netstat -u -l -p' shows that the recv queue averaging at around 40K.

Hmm, proxying should be fast. You may want to check there are no DNS
lookups, for example, %C in log configuration, or something other that
slows down the process.

> On the back end radius, it is almost zero most of the time.

This would indicate the backends are able to keep up with incoming requests.

> However running 'netstat -u -s' shows a lot of packet receive errors on both proxy radius and backend radius.

These are 'packet receive errors' under UDP section? If the UDP queue
overflows, then you would see the counter value increasing. It could be
something else too if the queue is empty or near empty while the receive
error counter value increases.

> Have tried to increasing the udp buffer on the radius servers. On the backend radius, it does helps in reducing the packet receive errors but not much improvement on the proxy radius.

Increasing buffer size should help with sudden surges, as it seems to do
at least partly.

Thanks,
Heikki

> 
> Best Regards,
> Ryan
> 
> 
> On 02/11/2014 02:04 PM, Ryan wrote:
> 
>>/ Two proxy radius servers serving a number of NASes. The proxy radius
> />/ servers will forward requests based on realms to sets of backend radius
> />/ servers.
> />/ 
> />/ Recently there seems to be a surge in requests whereby logs on the proxy
> />/ radius shows that the backend radius are not responding in time. Timeout
> />/ set is 10 sec with 3 retries. At times, it result in the unknown reply
> />/ from the backend radius.
> /
> If there are multiple backends, you could try setting a separate OutPort
> for some or all of them. This forces radiusd to create a socket for each
> bind address and port pair allowing it to spred outgoing requests
> between multiple sockets.
> 
>>/ Logs on the backend radius does not show any issues.
> /
> You may still want to enable debug logging (Trace 4) with
> LogMicroseconds and see if a typical authentication takes longer than
> you would expect. For example, if you use SQL, you should check how long
> the SQL lookup takes.
> 
>>/ Does this means that the proxy radius can't handle the traffic? And more
> />/ proxy radius are required to handle requests?
> /
> Most likely the backends are having problems serving all the requests.
> Unknown replies could be for example, requests that have spent a lot of
> time in the OS's incoming UDP queue. When they eventually get processed
> by the backend Radiator, the frontend has already given up on the
> request it sent. The timeout is quite long, though, but I'd still take a
> closer look at the backend performance.
> 
> Try running 'netstat -u -l -p' on the backend and see what Recv-Q shows.
> This tells if packets are queued before being processed by radiusd.
> 
> Another thing to check is 'netstat -u -s'. If UDP 'packet receive
> errors' counter value grows, the OS may be dropping incoming requests
> because the socket queue is full.
> 
>>/ All radiuses are running latest release.
> /
> Please let us know if the above helps.
> 
> Thanks,
> Heikki
> 
> -- 
> Heikki Vatiainen <hvn at open.com.au <http://www.open.com.au/mailman/listinfo/radiator>>
> 
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
> Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
> TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
> DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS,
> NetWare etc.
> 
> 
> 
> _______________________________________________
> radiator mailing list
> radiator at open.com.au
> http://www.open.com.au/mailman/listinfo/radiator
> 


-- 
Heikki Vatiainen <hvn at open.com.au>

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS,
NetWare etc.