[RADIATOR] Radiator performance
Alex Massover
alex at jajah.com
Wed Feb 10 02:46:17 CST 2010
Hi!
After a several hours of processing tens of requests per second the memory usage grows from 700M to 4G and the server hangs.
I run Radiator with dispatcher with 32 instances doing AuthbySOAP.
Does it need more than 4G memory? What are the requirements?
If Radiator processes requests serially there should be no memory usage growth from request to request as far as I understand.
Maybe it's a memory leak?
> -----Original Message-----
> From: Hugh Irvine [mailto:hugh at open.com.au]
> Sent: יום ג 09 פברואר 2010 12:17
> To: Alex Massover
> Subject: Re: [RADIATOR] Radiator performance
>
>
> Hello Alex -
>
> Yes it makes sense to do what you describe, and as RadSec is a TCP
> protocol it will certainly help UDP problems.
>
> BTW - you should also run separate Radiator instances for
> authentication and accounting which will improve performance.
>
> regards
>
> Hugh
>
>
> On 9 Feb 2010, at 20:59, Alex Massover wrote:
>
> > Yes, I see. Also 'netstat -su' clearly shows dropped packets.
> >
> > I configured big UDP buffer (50M), and configured the farm of 32
> instances - looks like it handles the designed load.
> >
> >
> > I have another question to consult, please. Because of performance
> issues NAS software does not wait for accounting reply and does not try
> to resend it.
> >
> > So I afraid of dropped packets. Does RadSec supposed to help me with
> it?
> >
> > I want to put RadSec locally on each NAS, NAS will send udp
> accounting requests to RadSec via local interface, and RadSec will send
> accounting requests to Radiator via TCP in reliable way.
> >
> > Does it make sense? Will it also help with UDP buffer queue in
> Radiator?
> >
> >> -----Original Message-----
> >> From: Hugh Irvine [mailto:hugh at open.com.au]
> >> Sent: יום ו 05 פברואר 2010 00:58
> >> To: Alex Massover
> >> Cc: radiator at open.com.au
> >> Subject: Re: [RADIATOR] Radiator performance
> >>
> >>
> >> Hello Alex -
> >>
> >> This problem has been discussed numerous times on the mailing list.
> >>
> >> What tends to happen is Radiator is kept waiting by an external
> >> resource such as a SOAP interface, the UDP buffer queue that
> Radiator
> >> is listening on fills up and overflows, and the client NAS equipment
> >> see requests timing out and retransmit.
> >>
> >> The problem is almost always due to the delay caused by the external
> >> resource.
> >>
> >> As mentioned in my previous mail, the first thing to do is look at a
> >> trace 4 debug with LogMicroseconds so you can see exactly how long
> each
> >> processing step is taking.
> >>
> >> In your case it sounds like there is a ~50 millisecond response time
> to
> >> your SOAP queries (ie. 20 requests per second).
> >>
> >> Note this comment in the example "goodies/soap.cfg" configuration
> file:
> >>
> >>
> >> # Caution: the overheads of creating and processing
> >> SOAP requests
> >> # mean you will never get carrier class performance
> >> from AuthBy SOAP
> >>
> >>
> >> Also note that if Radiator is not seeing the requests at all, this
> is
> >> confirmation that the system UDP buffer queue has overflowed.
> >>
> >> regards
> >>
> >> Hugh
> >>
> >>
> >> On 5 Feb 2010, at 01:46, Alex Massover wrote:
> >>
> >>> Hi!
> >>>
> >>> But my problem is not about delay, it's about losing/not processing
> >> request.
> >>>
> >>> For example I captured with wireshark an accounting request came to
> >> the server but no evidence to this request in radiator log. Meaning
> >> Radiator just silently ignored this request.
> >>>
> >>> Farm didn't help, I experience the same behavior.
> >>>
> >>> Radiator reports that there's no dropped request:
> >>>
> >>> Reply-Message = "Radiator Radius server version 4.5.1"
> >>> Reply-Message = "Running on nj-radius-stg1.jajah.dublin since Thu
> >> Feb 4 12:19:53 2010"
> >>> Reply-Message = "6 Requests in the last second"
> >>> Reply-Message = "0 Access accepts"
> >>> Reply-Message = "0 Access challenges"
> >>> Reply-Message = "0 Access rejects"
> >>> Reply-Message = "0 Access requests"
> >>> Reply-Message = "9143 Accounting requests"
> >>> Reply-Message = "9143 Accounting responses"
> >>> Reply-Message = "0 Bad authenticators in authentication requests"
> >>> Reply-Message = "0 Bad authenticators in accounting requests"
> >>> Reply-Message = "0 Total Bad authenticators in requests"
> >>> Reply-Message = "0 Dropped access requests"
> >>> Reply-Message = "0 Dropped accounting requests"
> >>> Reply-Message = "0 Total dropped requests"
> >>> Reply-Message = "0 Duplicate access requests"
> >>> Reply-Message = "0 Duplicate accounting requests"
> >>> Reply-Message = "0 Total duplicate requests"
> >>> Reply-Message = "0 Malformed access requests"
> >>> Reply-Message = "0 Malformed accounting requests"
> >>> Reply-Message = "0 Total proxied requests with no reply"
> >>> Reply-Message = "0 Total proxied requests"
> >>> Reply-Message = "9143 Total requests"
> >>> Reply-Message = "0.162143349699178 Average response time"
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Hugh Irvine [mailto:hugh at open.com.au]
> >>>> Sent: יום ד 03 פברואר 2010 00:41
> >>>> To: Alex Massover
> >>>> Cc: radiator at open.com.au
> >>>> Subject: Re: [RADIATOR] Radiator performance
> >>>>
> >>>>
> >>>> Hello Alex -
> >>>>
> >>>> The best way to see what is happening with your authentications is
> >> to
> >>>> run Radiator with a Trace 4 debug and a LogMicroseconds logger
> >>>> (requires Time-Hires from CPAN) so you can see how long each
> >> processing
> >>>> step is taking.
> >>>>
> >>>> Something like this in your configuration file:
> >>>>
> >>>> .....
> >>>>
> >>>> Trace 4
> >>>>
> >>>> <Log FILE>
> >>>> Filename %L/microseconds-%Y-%m-%d.log
> >>>> # requires Time-Hires from CPAN
> >>>> LogMicroseconds
> >>>> Trace 4
> >>>> </Log>
> >>>>
> >>>> .....
> >>>>
> >>>> This will add a six digit microseconds offset in each debug
> >> timestamp
> >>>> so you can see where the delays are.
> >>>>
> >>>> Once we have the LogMicroseconds debug we can see exactly what is
> >>>> happening.
> >>>>
> >>>> regards
> >>>>
> >>>> Hugh
> >>>>
> >>>>
> >>>> On 3 Feb 2010, at 00:49, Alex Massover wrote:
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> What performance I can expect from single Radiator server?
> >>>>>
> >>>>> Currently I’m getting timeout from clients with more than 20
> >> requests
> >>>> per second, on local LAN, with very low CPU usage and load
> average,
> >> on
> >>>> VMware ESX, RHEL 5.
> >>>>>
> >>>>> Is it OK and I should get more servers, or it supposed to handle
> >> much
> >>>> more? 20 requests per second doesn’t sound a lot for me.
> >>>>>
> >>>>> My configuration is also very simple and SOAP endpoint always
> >> answers
> >>>> fast, no timeouts/rejects from there.
> >>>>>
> >>>>> <Realm DEFAULT>
> >>>>> <Log SYSLOG>
> >>>>> Facility local2
> >>>>> Trace 5
> >>>>> </Log>
> >>>>> AuthByPolicy ContinueUntilAccept
> >>>>>
> >>>>> <AuthBy SOAP>
> >>>>> # Fork
> >>>>> Timeout 5
> >>>>> Endpoint http://MSG-LB-BES-STG:80/SoapHandler.ashx
> >>>>> SOAPTrace result
> >>>>> SOAPTrace all
> >>>>> </AuthBy>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Best Regards,
> >>>>> Alex Massover
> >>>>> VoIP R&D TL
> >>>>> Jajah Inc.
> >>>>>
> >>>>>
> >>>>>
> >>>>> This mail was sent via Mail-SeCure System.
> >>>>> _______________________________________________
> >>>>> radiator mailing list
> >>>>> radiator at open.com.au
> >>>>> http://www.open.com.au/mailman/listinfo/radiator
> >>>>
> >>>>
> >>>>
> >>>> NB:
> >>>>
> >>>> Have you read the reference manual ("doc/ref.html")?
> >>>> Have you searched the mailing list archive
> >>>> (www.open.com.au/archives/radiator)?
> >>>> Have you had a quick look on Google (www.google.com)?
> >>>> Have you included a copy of your configuration file (no secrets),
> >>>> together with a trace 4 debug showing what is happening?
> >>>>
> >>>> --
> >>>> Radiator: the most portable, flexible and configurable RADIUS
> server
> >>>> anywhere. Available on *NIX, *BSD, Windows, MacOS X.
> >>>> Includes support for reliable RADIUS transport (RadSec),
> >>>> and DIAMETER translation agent.
> >>>> -
> >>>> Nets: internetwork inventory and management - graphical,
> extensible,
> >>>> flexible with hardware, software, platform and database
> >> independence.
> >>>> -
> >>>> CATool: Private Certificate Authority for Unix and Unix-like
> >> systems.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> This mail was received via Mail-SeCure System.
> >>>>
> >>>
> >>>
> >>> This mail was sent via Mail-SeCure System.
> >>>
> >>>
> >>> _______________________________________________
> >>> radiator mailing list
> >>> radiator at open.com.au
> >>> http://www.open.com.au/mailman/listinfo/radiator
> >>
> >>
> >>
> >> NB:
> >>
> >> Have you read the reference manual ("doc/ref.html")?
> >> Have you searched the mailing list archive
> >> (www.open.com.au/archives/radiator)?
> >> Have you had a quick look on Google (www.google.com)?
> >> Have you included a copy of your configuration file (no secrets),
> >> together with a trace 4 debug showing what is happening?
> >>
> >> --
> >> Radiator: the most portable, flexible and configurable RADIUS server
> >> anywhere. Available on *NIX, *BSD, Windows, MacOS X.
> >> Includes support for reliable RADIUS transport (RadSec),
> >> and DIAMETER translation agent.
> >> -
> >> Nets: internetwork inventory and management - graphical, extensible,
> >> flexible with hardware, software, platform and database
> independence.
> >> -
> >> CATool: Private Certificate Authority for Unix and Unix-like
> systems.
> >>
> >>
> >>
> >>
> >> This mail was received via Mail-SeCure System.
> >>
> >
> >
> > This mail was sent via Mail-SeCure System.
>
>
>
> NB:
>
> Have you read the reference manual ("doc/ref.html")?
> Have you searched the mailing list archive
> (www.open.com.au/archives/radiator)?
> Have you had a quick look on Google (www.google.com)?
> Have you included a copy of your configuration file (no secrets),
> together with a trace 4 debug showing what is happening?
>
> --
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. Available on *NIX, *BSD, Windows, MacOS X.
> Includes support for reliable RADIUS transport (RadSec),
> and DIAMETER translation agent.
> -
> Nets: internetwork inventory and management - graphical, extensible,
> flexible with hardware, software, platform and database independence.
> -
> CATool: Private Certificate Authority for Unix and Unix-like systems.
>
>
>
>
> This mail was received via Mail-SeCure System.
>
This mail was sent via Mail-SeCure System.
More information about the radiator
mailing list