(RADIATOR) Input queue size

[Hugh Irvine]

Hello Claudio, Hello Guðbjörn -

Comments below.

On 13/11/2003, at 1:18 PM, Claudio Lapidus wrote:

> Hello Guðbjörn
>
>> this may be unrelated, but I am interested to any and all tuning
>> listmembers have done in the OS for Radiator performance. We
>> are running two radiator servers with one proxy radiator in front
>> and a seperate sql machine and ldap machine.
>
> Fine, but what OS do you use? It might be interesting to have a 
> hardware
> summary too.
>

Yes its useful to know the hardware/software platform and the various 
versions of Perl, etc.

> [snip]
>
>> Lengthening the udp queues seems to really have adverse effects on
>> this situation. We have not really tried shortening the queue which
>> might really have even more adverse effects, without testing though
>> I can't tell.
>
> I can imagine that lengthening the queue only adds to the effect of the
> server processing "old" packets, i.e. packets whose original timer (at 
> the
> NAS) has already expired. The root problem is the mismatch between the 
> speed
> of the NAS sending packets and the server processing them. Probably is 
> worth
> trying to increase the timeout setting at the NAS, at least to diminish
> retransmissions (but beware of total authentication time then). A 
> quicker
> failover to a less loaded secondary might help too.
>

Claudio is correct, the usual cause of problems of this sort is the 
backend delay associated with querying the LDAP and/or SQL database. It 
is very helpful to look at a trace 4 debug with "LogMicroseconds" 
turned on (requires Time-HiRes from CPAN). This will show exactly how 
much time is being spent waiting for the queries to complete.

And you are correct in your observation that increasing the queue size 
can adversely affect performance due to the increased number of retry 
requests that build up in the queue.


>>
>> To counter this we have configured multiple instances of radiators
>> for authentication&authorization and accounting and instances for
>> seperate NAS's or NAS groups. This in effect simulates having a
>> threaded radiator to reduce the effect of this sequential processing.
>
> OK, but are you sure that the bottleneck is in at the Radiator level or
> might it be at the LDAP server? In the latter case it probably won't 
> be of
> much help anyway.
>

Correct again. We have observed these problems too, when parallel 
requests can also slow things down.

BTW - this is one of the strong arguments against a multi-threaded 
server, which may not help at all in some situations.

In general it is easier in the first instance to do what you have done 
with multiple instances and a front end load balancer.

Just out of interest the largest Radiator setup we are familiar with is 
using this architecture, with a load balancer feeding 6 Radiator hosts, 
each one with an authentication and an accounting instance. The backend 
is a *very* fast Oracle database server and the overall throughput has 
been tested to over 1200 radius requests per second.

>>
>> This has not seemed to be related to CPU load or network performance,
>> we have looked at these in detail.
>
> No, it's probably more I/O bound, (disk, I mean).
>

I would agree - again a trace 4 debug with LogMicroseconds will show us 
exactly what is happening.

>> If anyone has input on this issue or OS tuning for Radiator I'd love
>> to hear about it. Hope you understand my attempt to explain the above
>> scenario. Basically we have a pretty stable environment today, but
>> perhaps overly complex to manage because of the multiple instances.
>
> Back to my original question then, I'm struggling to measure the 
> effective
> length of the input queue in Solaris. Linux's netstat shows it 
> readily, and
> I remember Tru64 doing the same. But Solaris' netstat lacks this one,
> apparently. I'll have to continue my quest...
>

On this topic, have you checked the Sunfreeware site to see if there 
are any useful tools in this regard?

	www.sunfreeware.com

>
>> Hugh, is a "threaded" ldap handler on the horizon? Is this perl or
>> radiator related?
>

This topic comes up from time to time and the fundamental problem at 
the moment is that Perl itself does not currently have "production 
quality" threading support. This being the case, we have not pursued it 
actively. And note my previous comments about whether or not this would 
be a "good thing" in any case.

>> From my own corner, I wish it were possible to have more than one
> established connection with the SQL backend, so as to paralellize 
> requests
> to a certain degree. But yes, I suppose that means multithreading, and 
> AFAIK
> that's not possible under perl 5.6 nor 5.8 I think. Perhaps Perl 6 
> would do
> it?
>

As mentioned above, the easiest way to do this currently is with a load 
balancer (you could use the AuthBy ROUNDROBIN, VOLUMEBALANCE, 
LOADBALANCE modules) and multiple instances of Radiator. Note that in 
most cases, at least using one instance for authentication and another 
for accounting is a good first step.

We will continue to monitor the Perl support for multi-threading too, 
of course.

regards

Hugh


NB: have you included a copy of your configuration file (no secrets),
together with a trace 4 debug showing what is happening?

-- 
Radiator: the most portable, flexible and configurable RADIUS server
anywhere. Available on *NIX, *BSD, Windows, MacOS X.
-
Nets: internetwork inventory and management - graphical, extensible,
flexible with hardware, software, platform and database independence.
-
CATool: Private Certificate Authority for Unix and Unix-like systems.

===
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.