(RADIATOR) Re: High Volume RADIUS (Modified by Hugh Irvine)

Mon Jan 24 02:37:44 CST 2005

Hello Everyone -

There have been a number of discussions about rate limiting RADIUS 
requests on the list and as you know I have never been happy with the 
idea of "rate limiting" per se. In thinking about it yesterday I had a 
chat with Mike about a simple alternative approach which is outlined 
below.

Obviously additional reply attributes could be returned by the AuthBy 
INTERNAL, to limit access to a "walled garden" for example until the 
re-authentication took place. The "walled garden" could take any web 
access to an information page indicating "Service temporarily 
unavailable ..." or whatever.

We would be very interested in hearing what people think about the 
approach and also hearing about operational experience.

Please let us know how you get on.

regards

Hugh

Forwarded mail from Mike <mikem at open.com.au>:

Hello all,

Interesting idea from Hugh.

Here is a sample implementation. If it sees more than 100 requests in a
second, the surplus will be ACCEPTed with a Session-Timeout randomly 
selected
between 1 and 11 seconds, at which time it should re-authenticate.

This makes it a bit like the random backoff implemented in some 
media-access
protocols. If you got a large number of requests in one seconds, they 
would
effectivley be deferred and the real authenticaitons smeared out into 
the
following 10 seconds.

Some experimentation will be required to determine operational values 
for the
number of requests per second threshold and the random backoff deferral 
period.

Cheers.

Mike

<Realm DEFAULT>
        AuthByPolicy ContinueWhileIgnore
        # This hook limits how many requests per
        # second will be passed on to the
        # AuthBy RADIUS. The surplus will be
        # ACCEPTed with a Session-Timeout randomly
        # selected between 1 and 11 seconds, at
        # which time the client should re-authenticate.
        <AuthBy INTERNAL>
	       RequestHook sub {my $time = time;\
		if ($time == $main::last_throttle_time) \
		{ if (++$main::throttle_count > 100)\
		  {\
		    $_[0]->{rp}->add_attr('Session-Timeout', int(rand(10) + 1));\
		    return ($main::ACCEPT, 'Conditional flood control');\
		  }\
		}\
		else\
		{    $main::throttle_count = 0;\
		}\
		$main::last_throttle_time = $time;\
		return ($main::IGNORE, 'Continue to proxy');\
	      }
        </AuthBy>

	<AuthBy RADIUS>
		# Your info here:
		Host localhost
		AuthPort 1647
		AcctPort 1648
	</AuthBy>
</Realm>

On Monday 24 January 2005 16:32, Hugh Irvine wrote:
> Hello Richard -
>
> In thinking about this problem a bit more I was struck by an
> alternative approach that may be worth thinking about.
>
> The idea is to unconditionally accept radius requests over a certain
> preset number of requests per second, but with a random number of
> seconds in a Session-Timeout reply attribute which would have the
> effect of processing the immediate queue as quickly as possible and
> cause the sessions to re-authenticate over a much longer period of time
> when the requests would be dealt with normally. This would be very
> simple to implement with an AuthBy INTERNAL clause and a hook before
> the AuthBy RADIUS clause. The hook could keep track of the number of
> requests per second and only kick in once the threshold limit was
> reached.
>
> What do you think?
>
> In this situation of course you would then receive the corresponding
> accounting requests, but if you configure an authentication instance
> and an accounting instance on your Radiator hosts this will be less of
> an issue.
>
> Further comments below.
>
> On 24 Jan 2005, at 16:01, Richard Staniforth wrote:
>> On Mon, 24 Jan 2005, Hugh Irvine wrote:
>>> Hello Richard -
>>
>> Thanks for the swift response.
>>
>>> The usual approach is first of all to rate-limit the radius requests
>>> from the Cisco equipment which is now possible in IOS. Then you can
>>> use
>>> a load-balancer (either a hardware device, or Radiator configured 
>>> with
>>> an AuthBy LOADBALANCE clause) to distribute the radius requests 
>>> across
>>> some number of Radiator hosts (typically 4 to 6). If you use Radiator
>>> you can also use the "UseExtendedIds" parameter to enable the use of 
>>> a
>>> much larger counter instead of the Identifier in the protocol header
>>> (the Proxy-State attribute is used for this). See section 6.30.18 in
>>> the Radiator 3.11 reference manual ("doc/ref.html").
>>
>> I am planning to implement the "UseExtendedIds" between our RADIUS
>> servers
>> and out wholesale clients. This will allow us to handle large volumes
>> of
>> RADIUS traffic to our clients, we need to be able to have the same
>> flexibility from the Access servers.
>>
>> I will look into the rate-limit command on the Cisco and see if that 
>> is
>> will contribute to the solution.
>>
>>> If you can't rate-limit in the Cisco equipment, the next best thing 
>>> is
>>> to set up rate limiting at some point in the router network between
>>> the
>>> access equipment and the Radiator host(s). Note that if you use the
>>> "radius source-port extended" IOS command, you will also need to
>>> disable duplicate detection in Radiator by setting "DupInterval 0" in
>>> your Client clauses. The reason for this is due to the unbounded
>>> memory
>>> usage which ensues due to the Cisco constantly changing the source
>>> port. In my view Cisco's "solution" is less than ideal.
>>
>> Will adding the source port to the duplicate requests index check 
>> allow
>> you to tell that the packet is a new request instead of a duplicate. I
>> realise that this will only work in the cisco source-port case but
>> then I
>> can work on a hardware solution if the requests can be indentified
>> uniquely and not as duplicates.
>
> I have my doubts about this - my recollection is that source port /
> identifier are always incremented, even for "duplicates".
>
> In any case you will need to use DupInterval 0 to stop memory
> exhaustion because of the unbounded nature of the source port /
> identifier combination.
>
>> Right now the queues are the problem both on the Cisco itself and the
>> RADIUS server receiving it. I have considered setting up a AuthBy
>> LoadBalance proxy but that will not solve the problem with the Cisco
>> wrapping its identifiers and losing track of those requests.
>>
>> My gut feeling the rate-limiting will improve the situation but still
>> give
>> us a delay the return to service.
>
> Note that there will _always_ be a delay in return to service when
> dealing with many thousands of connections on a single device.
>
>>> Unfortunately there isn't really anything that can alleviate the
>>> crunch
>>> that comes with many thousands of DSL (or other) sessions all trying
>>> to
>>> re-establish subsequent to some sort of outage. All you can really do
>>> is try to smooth out the avalanche of radius requests to allow the
>>> authentication system to cope over the period of time that it takes
>>> for
>>> all the sessions to come back up - typically some (largish) number of
>>> minutes.
>>>
>>> BTW - you can also split your radius request processing by pointing
>>> your dialup equipment at one set of Radiator hosts and your DSL at
>>> another set of Radiator hosts.
>>
>> Dial up aint so much a problem, the DSL is the big one but it is a
>> possibility to protect the Dial up.
>
> Understood.
>
> regards
>
> Hugh
>
>
>
> NB:
>
> Have you read the reference manual ("doc/ref.html")?
> Have you searched the mailing list archive
> (www.open.com.au/archives/radiator)?
> Have you had a quick look on Google (www.google.com)?
> Have you included a copy of your configuration file (no secrets),
> together with a trace 4 debug showing what is happening?

-- 
Mike McCauley                               mikem at open.com.au
Open System Consultants Pty. Ltd            Unix, Perl, Motif, C++, WWW
9 Bulbul Place Currumbin Waters QLD 4223 Australia   
http://www.open.com.au
Phone +61 7 5598-7474                       Fax   +61 7 5598-7070

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP etc on Unix, Windows, MacOS etc.

--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.