(RADIATOR) results: benchmarking and performance: multiple instances

[Tariq Rashid]

All, your comments/observations or corrections of teh following brief
resultys and conclusions are desired. This set of tests addresses the
potential improvement in radiatir performance (as request rate increases, we
are trying to mitigate the effects of request rate spikes which appear to
knock the server out for abotu 3 minutes) ...

<graph is not attached to mailing list - email me if you would like a small
pdf>

-------

attached is a graph showing the effects of using multiple instances of
radiator on a server - with requests proxied to each one by a very minimal
proxy-radiator. the graph shows the responce times as the request rate
increases, for experiments with 1,2,3,4,5, and 6 instances of radiator
behind a proxy. 

the configuration uses the ROUNDROBIN algorithm - the LOADBALANCE and
BALANCEVOLUME methods should work better when the instances are situated on
different servers with different speeds - but in this case they actually
degrade performance slightly. 

there are three main configration options with the proxy->multi-instance
configuration: use the proxy-instance to synchronously forward the requests,
do it asynchronously (where it doesn't wait for a responce) and finally, to
fork the proxy-radiator instance for each request.

the results from these are:
	* asynchronously loses too many requests - especially at higher
traffic rates
		this appears to be a weakness of radiator, it was also seen
during previous work

	* synchronously does not lose requests, the performance is certainly
not bad
		(see graph)

	* forking - is dangerous! as experienced during earlier experiments,
forking
		leads to a very rapid consumption of memory and cpu
resources!
		as discussed during the discussion of earlier expts, forking
copies the
		process entirely, and so during a peak in radius requests,
the server
		will fork that many radiator processes (even if they are
only the proxy-radiator)
		and bring the server down. i had to reboot my test box.
forking itself, aside
		from the memory issue, is relatively expensive on BSD. 
		This was also the reason behind implementing a threaded
benchmarking tool.

		In short, the radiator documentation does not recommend the
use of fork except in very
		special cases where there are few requests and they take
more than a fraction of a second
		to complete at the backend.

		To avoid a server irrecoverably locking up, we can set a
limit to the number of
		processes allowed to the UID running radiator. However, this
doesn't solve the
		original problem of servicing many requests.

What does the graph tell us?

	* that increasing the number of radiator instances on the same box
		does not significantly improve performance.
	
	* The critical request rate (above which performance starts to
degrade)
		does not change significantly.
	
	* The rate at which performance does degrade can sometimes be better
than
		the single instance case. In practise, this varied too often
to be
		dependable. However, in almost all cases, multiple instances
degrade
		less rapidly than a single instance. However, adding more
instances does
		not improve this degradation.

We conclude:

	* additional radiator instances do improve slightly the degredation
of
		overall performance as request rate increases ... BUT...
	* the important critical point (above which service degrades) is not
affected
	* this is likely due to the same UDP queue filling as seen earlier -
this time
		the input queue of the proxy-radiator which first receives
the
		requests fills up -> forming a bottlneck to the other
instances.

I will look into this just a bit more before moving on ... to make sure I
haven't made an oversight - I'm surprised the performance hasn't improved
much more than it has...

Tariq

--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.