(RADIATOR) radiator server under high load problem

[Brian Morris]

Hi Hugh / Steve,

We also experienced a similar sort of problem under load and are currently
still investigating the problem although it now appears to be corrected...

Running Radiator (as Proxy) on Sun /Solaris which sends requests to another
Radiator server running on Win2k with the database on a separate SQL Server
box again.  Plenty of RAM / CPU / etc etc everywhere.

At trace level 3 we found that we would receive a request (start / stop /
alive) from the Sun box to the Win2K box, which would authenticate and reply
within a second back to the Sun box but the Sun box would send a second
request 5 or less seconds later (retry was set to 5).

Note - we are still investigating but have made the following changes....

(a) replace the Win2k server hardware completely - no change
(b) replace the SQL Server hardware completely - no change

but

(c) Force all NICs to 100Mbs FULL DUPLEX (not auto detect)
(d) Increase the speed of the link between the two radius servers (was
traffic shaped to 256k)

The number of timeout errors has almost been eliminated since making changes
c+d.

I hope this helps.  I will post further information if I find the exact
cause / solution :-)

Regards,  Brian.


----- Original Message ----- 
From: "Hugh Irvine" <hugh at open.com.au>
To: "Steve Wilson" <radiator at swsystem.yorks.com>
Cc: "Radiator" <radiator at open.com.au>
Sent: Tuesday, July 22, 2003 9:25 PM
Subject: Re: (RADIATOR) radiator server under high load problem


>
> Hello Steve -
>
> Thanks for your mail - this topic comes up fairly frequently on the
> list.
>
> I don't really have enough information to give you definitive answers,
> but in my experience performance problems are almost always due to
> back-end services such as SQL databases and/or LDAP servers. I suggest
> you use the LogMicroseconds parameter in your logging (requires
> Time-HiRes from CPAN) so you can see exactly how much time is being
> used for each processing step. You can also use our companion product
> "Radar" to connect to a running Radiator instance which will give you a
> wide range of statistics together with fine-grained debugging and
> logging facilities. (BTW - you can also connect to the same Monitor
> port that supports Radar and issue commands manually, but I recommend
> Radar).
>
> You can also run two instances of Radiator - one for authentication and
> one for accounting.
>
> It may also be that you are seeing a problem with radius "Identifiers"
> when doing a lot of proxying, and if this is the case I recommend
> upgrading to the latest Radiator 3.6 (plus patches) and using the
> "UseExtendedIds" parameter in the AuthBy RADIUS clauses.
>
> If you have any further questions, please send me a copy of the
> Radiator configuration file (no secrets) and an example trace 4 or 5
> debug with LogMicroseconds turned on and I will take a look.
>
> regards
>
> Hugh
>
>
> On Tuesday, Jul 22, 2003, at 20:37 Australia/Melbourne, Steve Wilson
> wrote:
>
> > Hi,
> >
> > We are currently in the process of migrating about 5 icradius servers
> > into 1
> > system, we are currently at the stage where everything (about 10
> > NAS's) use
> > our a central radiator server to authenticate, this then sends the
> > request
> > to the correct icradius server based on the realm. As this is under
> > testing
> > I'm currently running radiator at debug level 5 to be able to diagnose
> > any
> > problems immediately. A serious problem arose overnight where this
> > morning
> > there were approx 2000 users all trying to authenticate at the same
> > time.
> > This left the machine under high load and it was then discovered that
> > radiator was no seeing the return radius accept packets, but tcpdump
> > saw
> > them returning to the machine.
> >
> > What I need to know is what can be upgraded/improved to allow heavy
> > debugging while users are rapidly connecting, the current machine spec
> > is
> > 2x2G Pentium III ( 256k L2 cache ),1G RAM, 36G raid5 (adaptec i2o).
> > This machine is running linux (kernel 2.4.19-16mdksecure SMP) and
> > version
> > 3.3.1-1 rpm version of Radiator.
> >
> > Any advice appreciated ;)
> >
> > In the future the 5 "backend" radius servers will turn into ldap
> > auths, is
> > this the best option ? Everything will be doubled so we have
> > redundancy too.
> >
> > Regards
> >
> > Steve Wilson
> >
> > Senior Systems Administrator
> > Legend Internet Ltd.
> >
> >
> > ===
> > Archive at http://www.open.com.au/archives/radiator/
> > Announcements on radiator-announce at open.com.au
> > To unsubscribe, email 'majordomo at open.com.au' with
> > 'unsubscribe radiator' in the body of the message.
> >
> >
>
> NB: have you included a copy of your configuration file (no secrets),
> together with a trace 4 debug showing what is happening?
>
> -- 
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. Available on *NIX, *BSD, Windows 95/98/2000, NT, MacOS X.
> -
> Nets: internetwork inventory and management - graphical, extensible,
> flexible with hardware, software, platform and database independence.
>
> ===
> Archive at http://www.open.com.au/archives/radiator/
> Announcements on radiator-announce at open.com.au
> To unsubscribe, email 'majordomo at open.com.au' with
> 'unsubscribe radiator' in the body of the message.
>

===
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.