(RADIATOR) Redback is sending too many Access-Requests

Wed Feb 1 03:22:54 CST 2006

Mishar, Hugh,

We had similar Problems with CISCO Equipment here (and it was not CISCO
which caused the problem). It took a while to find out, but normally the
accounts that 'hammer' down the pipe are trying to get in with a wrong
password and/or wrong username. Customer side equipment not deactivated
is most of the troublebringers.

Most of the xDSL-Gear on client side has what we start to call here
'crazy hammers' (older ZyXEL P310, P324 do this, and ZyXEL did take
precautions for the newer models against this 'hammering'). If they are
declined, they relentlessly try again, without pause between the
attempts, with this attemps efffectively draining the resources on the
NAS, every one of them doing - in a way - his own DOS attack. If you
have said you have 50+ users, you have a nice DDOS-Attack running, by
your own people (or ex-customers).

A (partial) remedy for this is to make the delay time on the PPP in the
NAS longer, so effectively not ALLOWING another start of a new
PPP-Session. We have it on 15 seconds, and it seems to work fine (we
have about the same amount of DSL-Customers on the box like you have).
This also helps if the box has to be rebooted or DSLAM's are taken down
and up again, and the whole population wants to get back in...

Another way is to spread the load, so you have your accounting on the
OracleDB and the authentication on another DB, provided by the main
DB... 

Another way is also to change the type of Authentication... LDAP comes
to mind, 2 LDAP server are MUCH MORE responsive than any DB with
authenticating (you are pulling too much overhead with
DB-Authenticating. It's ok to do Authentication by SQL, but the
responsiveness has a sharp drop if you are over a distinct amounts of
transactions per second...)

The ultimate Remedy (but your Support won't love you for this) is, to
find out WHO and WHY these 50+ 'bad' clients are there hammering the
NAS... In my case it was the ultimate 'cleanup', Support didn't talk
with Systems for 2 weeks :-), but they called all and everyone who's
equipment was hammering in ... contact all ex-users and bring them to
deactivate the still running equipment. Bring the Telco to take down the
DSL and so on...  poof.... no more hammering... 

Martin Wallner

________________________________

	From: owner-radiator at open.com.au
[mailto:owner-radiator at open.com.au] On Behalf Of Mishari Al-Faris
	Sent: Mittwoch, 01. Februar 2006 08:20
	To: Hugh Irvine
	Cc: radiator at open.com.au
	Subject: Re: (RADIATOR) Redback is sending too many
Access-Requests

	Hello Hugh,

	I understand that it will "accept" and spread them randomly over
1-20 mins. That is fine with me, as long as I treat the offending users
differently than the normal ones. My problem is that my oracle DB behind
radiator can not handle all the requests, the good and the bad ones. The
bad requests are relentless and can constitute over +95% of the incoming
requests. I have about 7500+ DSL customers and they stop complaining
once I manually do "AuthBy AllowAll" temporarily, however, I have around
200+ customers who require static IPs obtained from the DB, and once I
turn off authentication those static-IP customers start getting dynamic
IPs and start complaining, so I can't stop authentication for long.

	The amount of usernames that produce the bad requests is very
small compared to the total amount of users, say around 50+. And those
are the ones I want to exclude, also, I'm not sure what's common between
them that causes them to send so many requests. So far what I see from
these users is that they send 1 Access-Request every 1-5 seconds,
consistently and without stopping, regardless of what response I send
them, Accept or Reject doesn't matter, they keep on sending. And even
when I use AllowAll, which is the fastest response I can achieve, they
still send the requests, so I'm ruling out that the radius is slow to
respond to them.

	Also, the requests that come from the same user each has a
different "Acct-Session-Id". And the Redback is configured for 60
seconds timeout and only 1 retry. So this makes me rule out that the
Redback is retrying for the same user. Instead what I think is happening
is that the user connection to the Redback is unstable and disconnecting
and connecting again quickly, which causes the Redback to create a new
request for him.

	So, I'm willing to try this solution even if the per-user
counters take up alot of memory, if it doesn't work I'll stop using it,
but in the meantime I'm in trouble and am willing to try anything :(

	Ofcourse it goes without saying that I'm not holding any
guarantees against anyone either. If it works it works.

	So what I want to do is to exclude the few users that send that
many requests and either Accept them, or Ignore them, or Reject them,
doesnt really matter, as long as I don't go to the AuthBy PLSQL clause
for these guys. hence protecting my DB from the onslaught.

	On 2/1/06, Hugh Irvine <hugh at open.com.au> wrote: 

		Hello Mishari -

		You should note first of all that this hook code will
not "ignore" 
		the access requests - the hook is designed to "accept"
all access
		requests over a certain number with a variable session
timeout that
		will cause the resulting temporary session to drop after
some random 
		time. The idea being to spread the requests over a
longer period of
		time. Also note that this is an idea only - I make no
guarantees
		about the success or otherwise of using this code.

		I am also not sure about maintaining per-user counters,
as this will 
		lead to greatly increased memory usage.

		Can you tell me exactly what you are wanting to do?

		regards

		Hugh

		On 31 Jan 2006, at 22:53, Mishari Al-Faris wrote:

		> Dear Hugh,
		>
		> This is an example that you suggested a while back to
mitigate
		> excessive requests coming from DSL NASes.
		> I've been trying to modify it to our needs but have
been getting
		> compilation errors. Let me just explain what I wish to
do instead 
		> of going through what I did wrong.
		>
		> I'd like to count the access trials per "user" not per
"NAS". If a
		> certain username is seen trying more than say 1
request per 5
		> seconds, I want to ignore the request, and not go
through my 
		> AuthPLSQL AuthBy clause. Is this possible? thanks.
		>
		> # RequestHook for AuthBy INTERNAL
		> # This hook counts the number of access requests that
are received
		> for a
		> # particular NAS, and returns an ACCEPT if there are
more than 100 
		> per second.
		> # A Session-Timeout reply attribute is added to the
reply with a
		> random
		> # value between 1 and 1200 seconds(20 minutes).
		> #
		> # Note: these values should be altered as required. 
		> #
		> # Hugh Irvine, Open System Consultants, 20050829
		>
		> sub
		> {
		> my $p = $_[0];
		> my $time = time;
		> my $code = $p->code;
		> my $nas = $p->{Client};
		> if ($time == $nas->{last_throttle_time} && $code eq
'Access- Request') 
		> {
		> if (++$nas->{throttle_count} > 100)
		> {
		> $p->{rp}->add_attr('Session-Timeout', int (rand(1200)
+ 1));
		> return ($main::ACCEPT, 'Conditional flood control');
		> }
		> } 
		> else
		> {
		> $nas->{throttle_count} = 0;
		> }
		> $nas->{last_throttle_time} = $time;
		> return ($main::IGNORE, 'Continue to proxy');
		> }
		>
		>
		>
		> Here is an example of how to use the hook. 
		>
		>
		> <Handler .....>
		>
		> AuthByPolicy ContinueWhileIgnore
		>
		> <AuthBy INTERNAL>
		> RequestHook file:"throttle.pl"
		> AddToReply .....
		> </AuthBy> 
		> # normal AuthBy
		> <AuthBy .....>
		> .....
		> </AuthBy>
		> </Handler>

		NB:

		Have you read the reference manual ("doc/ref.html")?
		Have you searched the mailing list archive (
www.open.com.au/archives/
		radiator)?
		Have you had a quick look on Google (www.google.com)?
		Have you included a copy of your configuration file (no
secrets), 
		together with a trace 4 debug showing what is happening?

		--
		Radiator: the most portable, flexible and configurable
RADIUS server
		anywhere. Available on *NIX, *BSD, Windows, MacOS X.
		-
		Nets: internetwork inventory and management - graphical,
extensible, 
		flexible with hardware, software, platform and database
independence.
		-
		CATool: Private Certificate Authority for Unix and
Unix-like systems.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.open.com.au/pipermail/radiator/attachments/20060201/47cd0761/attachment.html>