[RADIATOR] Radiator performance
Alex Massover
alex at jajah.com
Wed Feb 10 06:30:26 CST 2010
Hi,
At first sight looks like it helped a lot, now Radiator instances are stable at ~23M each.
NoIgnoreDuplicates tends to be a dangerous option ;)
Thanks!
> -----Original Message-----
> From: Francisco Rodrigo Cortinas Maseda
> [mailto:francisco.cortinas at jazztel.com]
> Sent: יום ד 10 פברואר 2010 13:43
> To: Alex Massover; Hugh Irvine
> Cc: radiator at open.com.au
> Subject: RE: [RADIATOR] Radiator performance
>
> Hi Alex,
>
> Try:
>
> <Client DEFAULT>
> Secret mysecret
> # DupInterval 0
> # NoIgnoreDuplicates Accounting-Request
> </Client>
>
>
> Francisco Rodrigo Cortiñas
> Jazz Telecom SAU
> Telefono: +34 91 183 9046
> Email: francisco.cortinas at jazztel.com
>
>
> -----Mensaje original-----
> De: radiator-bounces at open.com.au [mailto:radiator-bounces at open.com.au]
> En nombre de Alex Massover
> Enviado el: miércoles 10 de febrero de 2010 12:18
> Para: Hugh Irvine
> CC: radiator at open.com.au
> Asunto: Re: [RADIATOR] Radiator performance
>
>
> Hi,
>
> And this still happens:
> a) without syslog (with Trace 0)
> b) without Farm
> c) without SOAP (with AuthByInternal)
>
> It really looks like every request came is not getting freed from the
> memory (I only test accounting requests). I'm using Radiator 4.5.1 on
> Red Hat Enterprise Linux Server release 5.2 (Tikanga) x86_64
>
>
> Here's my full config:
>
> FarmSize 32
> SocketQueueLength 524288000
> #Foreground
> #LogStdout
> LogDir /var/log/radius
> DbDir /etc/radiator
> DictionaryFile %D/dictionary, %D/dictionary.sip
> # Use a low trace level in production systems. Increase
> # it to 4 or 5 for debugging, or use the -trace flag to radiusd
> Trace 0
>
>
> # You will probably want to add other Clients to suit your site, # one
> for each NAS you want to work with <Client DEFAULT>
> Secret mysecret
> # DupInterval 0
> NoIgnoreDuplicates Accounting-Request
> </Client>
> <Realm loadbalancer>
> <AuthBy INTERNAL>
> DefaultResult ACCEPT
> </AuthBy>
> </Realm>
>
> <Realm DEFAULT>
> <Log SYSLOG>
> Facility local2
> Trace 5
> </Log>
>
> <AuthBy SOAP>
> Timeout 1
> Endpoint http://MSG-LB-BES-STG:80/SoapHandler.ashx
> SOAPTrace all
> </AuthBy>
>
> </Realm>
>
> > -----Original Message-----
> > From: Alex Massover
> > Sent: יום ד 10 פברואר 2010 10:46
> > To: 'Hugh Irvine'
> > Cc: radiator at open.com.au
> > Subject: RE: [RADIATOR] Radiator performance
> >
> > Hi!
> >
> > After a several hours of processing tens of requests per second the
> > memory usage grows from 700M to 4G and the server hangs.
> >
> > I run Radiator with dispatcher with 32 instances doing AuthbySOAP.
> >
> > Does it need more than 4G memory? What are the requirements?
> >
> > If Radiator processes requests serially there should be no memory
> > usage growth from request to request as far as I understand.
> >
> > Maybe it's a memory leak?
> >
> > > -----Original Message-----
> > > From: Hugh Irvine [mailto:hugh at open.com.au]
> > > Sent: יום ג 09 פברואר 2010 12:17
> > > To: Alex Massover
> > > Subject: Re: [RADIATOR] Radiator performance
> > >
> > >
> > > Hello Alex -
> > >
> > > Yes it makes sense to do what you describe, and as RadSec is a TCP
> > > protocol it will certainly help UDP problems.
> > >
> > > BTW - you should also run separate Radiator instances for
> > > authentication and accounting which will improve performance.
> > >
> > > regards
> > >
> > > Hugh
> > >
> > >
> > > On 9 Feb 2010, at 20:59, Alex Massover wrote:
> > >
> > > > Yes, I see. Also 'netstat -su' clearly shows dropped packets.
> > > >
> > > > I configured big UDP buffer (50M), and configured the farm of 32
> > > instances - looks like it handles the designed load.
> > > >
> > > >
> > > > I have another question to consult, please. Because of
> performance
> > > issues NAS software does not wait for accounting reply and does not
> > try
> > > to resend it.
> > > >
> > > > So I afraid of dropped packets. Does RadSec supposed to help me
> > with
> > > it?
> > > >
> > > > I want to put RadSec locally on each NAS, NAS will send udp
> > > accounting requests to RadSec via local interface, and RadSec will
> > send
> > > accounting requests to Radiator via TCP in reliable way.
> > > >
> > > > Does it make sense? Will it also help with UDP buffer queue in
> > > Radiator?
> > > >
> > > >> -----Original Message-----
> > > >> From: Hugh Irvine [mailto:hugh at open.com.au]
> > > >> Sent: יום ו 05 פברואר 2010 00:58
> > > >> To: Alex Massover
> > > >> Cc: radiator at open.com.au
> > > >> Subject: Re: [RADIATOR] Radiator performance
> > > >>
> > > >>
> > > >> Hello Alex -
> > > >>
> > > >> This problem has been discussed numerous times on the mailing
> > list.
> > > >>
> > > >> What tends to happen is Radiator is kept waiting by an external
> > > >> resource such as a SOAP interface, the UDP buffer queue that
> > > Radiator
> > > >> is listening on fills up and overflows, and the client NAS
> > equipment
> > > >> see requests timing out and retransmit.
> > > >>
> > > >> The problem is almost always due to the delay caused by the
> > external
> > > >> resource.
> > > >>
> > > >> As mentioned in my previous mail, the first thing to do is look
> > > >> at
> > a
> > > >> trace 4 debug with LogMicroseconds so you can see exactly how
> > > >> long
> > > each
> > > >> processing step is taking.
> > > >>
> > > >> In your case it sounds like there is a ~50 millisecond response
> > time
> > > to
> > > >> your SOAP queries (ie. 20 requests per second).
> > > >>
> > > >> Note this comment in the example "goodies/soap.cfg"
> configuration
> > > file:
> > > >>
> > > >>
> > > >> # Caution: the overheads of creating and
> > > >> processing SOAP requests
> > > >> # mean you will never get carrier class
> > > >> performance from AuthBy SOAP
> > > >>
> > > >>
> > > >> Also note that if Radiator is not seeing the requests at all,
> > > >> this
> > > is
> > > >> confirmation that the system UDP buffer queue has overflowed.
> > > >>
> > > >> regards
> > > >>
> > > >> Hugh
> > > >>
> > > >>
> > > >> On 5 Feb 2010, at 01:46, Alex Massover wrote:
> > > >>
> > > >>> Hi!
> > > >>>
> > > >>> But my problem is not about delay, it's about losing/not
> > processing
> > > >> request.
> > > >>>
> > > >>> For example I captured with wireshark an accounting request
> came
> > to
> > > >> the server but no evidence to this request in radiator log.
> > Meaning
> > > >> Radiator just silently ignored this request.
> > > >>>
> > > >>> Farm didn't help, I experience the same behavior.
> > > >>>
> > > >>> Radiator reports that there's no dropped request:
> > > >>>
> > > >>> Reply-Message = "Radiator Radius server version 4.5.1"
> > > >>> Reply-Message = "Running on nj-radius-stg1.jajah.dublin
> > since Thu
> > > >> Feb 4 12:19:53 2010"
> > > >>> Reply-Message = "6 Requests in the last second"
> > > >>> Reply-Message = "0 Access accepts"
> > > >>> Reply-Message = "0 Access challenges"
> > > >>> Reply-Message = "0 Access rejects"
> > > >>> Reply-Message = "0 Access requests"
> > > >>> Reply-Message = "9143 Accounting requests"
> > > >>> Reply-Message = "9143 Accounting responses"
> > > >>> Reply-Message = "0 Bad authenticators in authentication
> > requests"
> > > >>> Reply-Message = "0 Bad authenticators in accounting
> > requests"
> > > >>> Reply-Message = "0 Total Bad authenticators in
> requests"
> > > >>> Reply-Message = "0 Dropped access requests"
> > > >>> Reply-Message = "0 Dropped accounting requests"
> > > >>> Reply-Message = "0 Total dropped requests"
> > > >>> Reply-Message = "0 Duplicate access requests"
> > > >>> Reply-Message = "0 Duplicate accounting requests"
> > > >>> Reply-Message = "0 Total duplicate requests"
> > > >>> Reply-Message = "0 Malformed access requests"
> > > >>> Reply-Message = "0 Malformed accounting requests"
> > > >>> Reply-Message = "0 Total proxied requests with no
> reply"
> > > >>> Reply-Message = "0 Total proxied requests"
> > > >>> Reply-Message = "9143 Total requests"
> > > >>> Reply-Message = "0.162143349699178 Average response
> > > >>> time"
> > > >>>
> > > >>>
> > > >>>> -----Original Message-----
> > > >>>> From: Hugh Irvine [mailto:hugh at open.com.au]
> > > >>>> Sent: יום ד 03 פברואר 2010 00:41
> > > >>>> To: Alex Massover
> > > >>>> Cc: radiator at open.com.au
> > > >>>> Subject: Re: [RADIATOR] Radiator performance
> > > >>>>
> > > >>>>
> > > >>>> Hello Alex -
> > > >>>>
> > > >>>> The best way to see what is happening with your
> authentications
> > is
> > > >> to
> > > >>>> run Radiator with a Trace 4 debug and a LogMicroseconds logger
> > > >>>> (requires Time-Hires from CPAN) so you can see how long each
> > > >> processing
> > > >>>> step is taking.
> > > >>>>
> > > >>>> Something like this in your configuration file:
> > > >>>>
> > > >>>> .....
> > > >>>>
> > > >>>> Trace 4
> > > >>>>
> > > >>>> <Log FILE>
> > > >>>> Filename %L/microseconds-%Y-%m-%d.log
> > > >>>> # requires Time-Hires from CPAN
> > > >>>> LogMicroseconds
> > > >>>> Trace 4
> > > >>>> </Log>
> > > >>>>
> > > >>>> .....
> > > >>>>
> > > >>>> This will add a six digit microseconds offset in each debug
> > > >> timestamp
> > > >>>> so you can see where the delays are.
> > > >>>>
> > > >>>> Once we have the LogMicroseconds debug we can see exactly what
> > is
> > > >>>> happening.
> > > >>>>
> > > >>>> regards
> > > >>>>
> > > >>>> Hugh
> > > >>>>
> > > >>>>
> > > >>>> On 3 Feb 2010, at 00:49, Alex Massover wrote:
> > > >>>>
> > > >>>>> Hi!
> > > >>>>>
> > > >>>>> What performance I can expect from single Radiator server?
> > > >>>>>
> > > >>>>> Currently I’m getting timeout from clients with more than 20
> > > >> requests
> > > >>>> per second, on local LAN, with very low CPU usage and load
> > > average,
> > > >> on
> > > >>>> VMware ESX, RHEL 5.
> > > >>>>>
> > > >>>>> Is it OK and I should get more servers, or it supposed to
> > handle
> > > >> much
> > > >>>> more? 20 requests per second doesn’t sound a lot for me.
> > > >>>>>
> > > >>>>> My configuration is also very simple and SOAP endpoint always
> > > >> answers
> > > >>>> fast, no timeouts/rejects from there.
> > > >>>>>
> > > >>>>> <Realm DEFAULT>
> > > >>>>> <Log SYSLOG>
> > > >>>>> Facility local2
> > > >>>>> Trace 5
> > > >>>>> </Log>
> > > >>>>> AuthByPolicy ContinueUntilAccept
> > > >>>>>
> > > >>>>> <AuthBy SOAP>
> > > >>>>> # Fork
> > > >>>>> Timeout 5
> > > >>>>> Endpoint http://MSG-LB-BES-
> STG:80/SoapHandler.ashx
> > > >>>>> SOAPTrace result
> > > >>>>> SOAPTrace all
> > > >>>>> </AuthBy>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> Best Regards,
> > > >>>>> Alex Massover
> > > >>>>> VoIP R&D TL
> > > >>>>> Jajah Inc.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> This mail was sent via Mail-SeCure System.
> > > >>>>> _______________________________________________
> > > >>>>> radiator mailing list
> > > >>>>> radiator at open.com.au
> > > >>>>> http://www.open.com.au/mailman/listinfo/radiator
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> NB:
> > > >>>>
> > > >>>> Have you read the reference manual ("doc/ref.html")? Have you
> > > >>>> searched the mailing list archive
> > > >>>> (www.open.com.au/archives/radiator)?
> > > >>>> Have you had a quick look on Google (www.google.com)? Have you
> > > >>>> included a copy of your configuration file (no
> > secrets),
> > > >>>> together with a trace 4 debug showing what is happening?
> > > >>>>
> > > >>>> --
> > > >>>> Radiator: the most portable, flexible and configurable RADIUS
> > > server
> > > >>>> anywhere. Available on *NIX, *BSD, Windows, MacOS X. Includes
> > > >>>> support for reliable RADIUS transport (RadSec), and DIAMETER
> > > >>>> translation agent.
> > > >>>> -
> > > >>>> Nets: internetwork inventory and management - graphical,
> > > extensible,
> > > >>>> flexible with hardware, software, platform and database
> > > >> independence.
> > > >>>> -
> > > >>>> CATool: Private Certificate Authority for Unix and Unix-like
> > > >> systems.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> This mail was received via Mail-SeCure System.
> > > >>>>
> > > >>>
> > > >>>
> > > >>> This mail was sent via Mail-SeCure System.
> > > >>>
> > > >>>
> > > >>> _______________________________________________
> > > >>> radiator mailing list
> > > >>> radiator at open.com.au
> > > >>> http://www.open.com.au/mailman/listinfo/radiator
> > > >>
> > > >>
> > > >>
> > > >> NB:
> > > >>
> > > >> Have you read the reference manual ("doc/ref.html")? Have you
> > > >> searched the mailing list archive
> > > >> (www.open.com.au/archives/radiator)?
> > > >> Have you had a quick look on Google (www.google.com)? Have you
> > > >> included a copy of your configuration file (no secrets),
> together
> > > >> with a trace 4 debug showing what is happening?
> > > >>
> > > >> --
> > > >> Radiator: the most portable, flexible and configurable RADIUS
> > server
> > > >> anywhere. Available on *NIX, *BSD, Windows, MacOS X. Includes
> > > >> support for reliable RADIUS transport (RadSec), and DIAMETER
> > > >> translation agent.
> > > >> -
> > > >> Nets: internetwork inventory and management - graphical,
> > extensible,
> > > >> flexible with hardware, software, platform and database
> > > independence.
> > > >> -
> > > >> CATool: Private Certificate Authority for Unix and Unix-like
> > > systems.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> This mail was received via Mail-SeCure System.
> > > >>
> > > >
> > > >
> > > > This mail was sent via Mail-SeCure System.
> > >
> > >
> > >
> > > NB:
> > >
> > > Have you read the reference manual ("doc/ref.html")?
> > > Have you searched the mailing list archive
> > > (www.open.com.au/archives/radiator)?
> > > Have you had a quick look on Google (www.google.com)?
> > > Have you included a copy of your configuration file (no secrets),
> > > together with a trace 4 debug showing what is happening?
> > >
> > > --
> > > Radiator: the most portable, flexible and configurable RADIUS
> server
> > > anywhere. Available on *NIX, *BSD, Windows, MacOS X. Includes
> > > support for reliable RADIUS transport (RadSec), and DIAMETER
> > > translation agent.
> > > -
> > > Nets: internetwork inventory and management - graphical,
> extensible,
> > > flexible with hardware, software, platform and database
> > > independence.
> > > -
> > > CATool: Private Certificate Authority for Unix and Unix-like
> systems.
> > >
> > >
> > >
> > >
> > > This mail was received via Mail-SeCure System.
> > >
>
>
> This mail was sent via Mail-SeCure System.
> _______________________________________________
> radiator mailing list
> radiator at open.com.au http://www.open.com.au/mailman/listinfo/radiator
>
> Antes de imprimir este e-mail piense bien si es necesario hacerlo.
>
>
> -----------------------------------------------------------------------
> ---------
>
> Este mensaje es privado y CONFIDENCIAL y se dirige exclusivamente a su
> destinatario. Si usted ha recibido este mensaje por error, no debe
> revelar, copiar, distribuir o usarlo en ningún sentido. Le rogamos lo
> comunique al remitente y borre dicho mensaje y cualquier documento
> adjunto que pudiera contener. El correo electrónico via Internet no
> permite asegurar la confidencialidad de los mensajes que se transmiten
> ni su integridad o correcta recepción. JAZZTEL no asume responsabilidad
> por estas circunstancias. Si el destinatario de este mensaje no
> consintiera la utilización del correo electrónico via Internet y la
> grabación de los mensajes, rogamos lo ponga en nuestro conocimiento de
> forma inmediata.Cualquier opinión expresada en este mensaje pertenece
> únicamente al autor remitente, y no representa necesariamente la
> opinión de JAZZTEL, a no ser que expresamente se diga y el remitente
> esté autorizado para hacerlo.
>
>
> -----------------------------------------------------------------------
> ---------
>
>
> This message is private and CONFIDENTIAL and it is intended exclusively
> for its addressee. If you receive this message in error, you should not
> disclose, copy, distribute this e-mail or use it in any other way.
> Please inform the sender and delete the message and attachments from
> your system.Internet e-mail neither guarantees the confidentiality nor
> the integrity or proper receipt of the messages sent. JAZZTEL does not
> assume any liability for those circumstances. If the addressee of this
> message does not consent to the use of Internet e-mail and message
> recording, please notify us immediately.Any views or opinions contained
> in this message are solely those of the author, and do not necessarily
> represent those of JAZZTEL, unless otherwise specifically stated and
> the sender is authorised to do so.
>
>
> -----------------------------------------------------------------------
> ---------
>
> This mail was received via Mail-SeCure System.
This mail was sent via Mail-SeCure System.
More information about the radiator
mailing list