(RADIATOR) Threading support (again)

Mon Sep 3 11:05:49 CDT 2007

Hi!

Same problems here (peaks and so on)... where I'm not thinking that -
with current hardware - the problem does not lie in the amount of
requests and records that are sent to RADIATOR, the problem lies in the
authentication and accounting backbone of the system.

Accounting inserts sometimes (and especially with excessive need of
re-writing the indicies) are a terrible strain on the DB, which then
goes back to Radiator, which is waiting for the ack to the db, that
sometime goes in to timeout, and so on.

What's also straining is the amount of Alives, that starts to mount up
when you are getting more and more users.
The only valid and usable method to store the Accounting Infos is in SQL
(sooner or later it goes in some form of DB anyway, so why handling and
transforming GIGANTIC text files in the DB when one can put it in the db
directly)

Currently in the 'goodies', the SQL preposition is to write the
SQL-Accounting table like a text file, just slop the Accounting Data in
the table and be done with it (which makes for a LOT of WASTE in the
Accounting DB, especially, because normally your accounting dept. is
interested in either the last Alive record or the Stop record). This
makes f.e. our current setup write up to 1 Mil+ entries in the DB within
24 hours.... I am NOT criticising the way it's done in 'goodies' but
it's for low to medium high usage. The database server can also start to
'split' up the data to make something like getting data for usage for
the customer easier and so on... Basically, that means, that per day and
session a maximum of 3 entries (Start - 1 time, Alive - n times updated,
and Stop - 1 time)... also, it would not need the Index to become
monsterous... (we are just testing this here and we get around 25% of
the Records instead of the 1 Mil +....) and you can do nice
preprocessing in the SQL Server of the data... Maybe I can get my bosses
to put this into RADIATOR-pd... 

Another way - maybe - is to make more than one connections to the
Database; f.e. make more SQL-Authentication Clauses for the different
Realms or handlers, bringing Radiator to open up more DB-Connections
(never tested this...)

I'm not thinking that asynchronous DBI will help here much, because the
strain is still here, and, even worse, Radiator wouldn't be able to
clean out the complete transaction, he would have to store it until it
gots acked or Radiator calls for the ack... 

Additionally I found out that the current way of doing the
radonline-table is also making up a lot of clutter in the index and (I'm
using PostGreSQL) running the vacuum daemon is mandatory...

So, in conclusion, besides from running up to 4 Radiators (2 Accounting
/ 2 Authenticatings) on a 4 Core Server, doing LDAP Authentication and
SQL Accounting, we should first and furthermore see what we can do at
the Interfaces and the Handling of the queues to the backbone services
and get this more streamlined. The current Radiator Core is (on a 4 Core
HP ProLiant 360 G4p ) good for 200-300 Transactions per second (all
local userdb and local accounting file).... The moment one tries to make
the work easier by bringing in more sophisticated stuff, it can get
ugly... 

Martin Wallner
IP/Systems sen. Engineer 
eTel Austria AG, Haydngasse 17, A-1060 Wien
Tel.:      +43 (0)50 1011 - 2254
Mobil:    +43 (0)699 1599 - 2254
E-Mail:  martin.wallner at etel.at         
www.etel.at

-----Original Message-----
From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au] On
Behalf Of Hartmaier Alexander
Sent: Montag, 03. September 2007 15:55
To: radiator at open.com.au
Subject: RE: (RADIATOR) Threading support (again)

Hi!

We encountered similar problems in the last few month when traffic
rises.
Have you thought about using POE and various components (like
asynchronous DBI
calls) for Radiator?
For us it seems that the accounting inserts block radiator sometimes.
How could be snmp checks be logged to verify that aren't the cause of
our problems?
I never saw a single log line for those even with a trace 5 log.

With best regards
Alexander Hartmaier

T-Systems Austria GesmbH
Rennweg 97-99
A-1030 Vienna

phone: +43-(0)57057-4320
mobile: +43-(0)676-8642-4320
-----Original Message-----
From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au] On
Behalf Of John Morrissey
Sent: Friday, August 24, 2007 11:46 PM
To: radiator at open.com.au
Subject: (RADIATOR) Threading support (again)

Looking through the archives, it's been a long time since someone's
kicked the threading horse.

We've been running Radiator on a fairly substantial RADIUS installation
(a dozen or so machines in three locations across the United States) for
several years now, making use of an LDAP directory for account storage.
(Hugh or Mike, feel free to e-mail me privately if you need more
information.) Unfortunately, OpenLDAP libraries do not seem well behaved
when Radiator forks, although it's been a while since we've tried.

Our largest problem with a monolithic process is that we've had
occasional problems where device outages cause thousands of subscribers
to reauthenticate simultaneously. This overwhelms our authentication
radiusd instances, causing UDP network queues on that port to skyrocket,
eventually leading to a cascade failure that is obnoxious to recover
from, to say the least. Our RADIUS servers are by no means underpowered,
but it's just not cost-effective to maintain a huge (and otherwise idle)
hardware pool just to handle occasional huge spikes like this.

Additionally, we're starting to deploy machines with four CPU cores as
their price has dropped substantially. We like sticking to a fairly
standard configuration, so we'd like to avoid "wasting" hardware that
Radiator will never use (we have auth and accounting split into two
Radiator instances).
We could run more radiusd processes on different ports or IP addresses,
but this seems wasteful and complex.

Lastly, we're looking at implementing simultaneous use restrictions and
are concerned that the state refresh (SNMP in some cases, locally
implemented interfaces to proprietary devices in others) will block
radiusd excessively.
We're also looking at proxying to more customers, and are concerned that
unreachable or underperforming proxy destinations will cause the same
behavior.

We had considered implementing rate limiting to handle our first concern
above, but that doesn't get us anywhere with the others. To that end,
I'm toying with the idea of adding a new process model to Radiator
somewhat akin to Apache's prefork MPM, with a number of child processes
(not threads) running round-robin on the incoming UDP port or handling
work directed at them via some sort of IPC from a central request
handler process.

I haven't dived into the code yet to see how non-trivial this will be,
and I thought I'd ask around here first to see what everyone's thoughts
are. I know that a threaded model hasn't been implemented due to
alpha-quality thread support in Perl, and I haven't looked into its
current state yet.
That's part of the reason I'm keen on using separate worker processes
instead of threads.

thanks,
john
-- 
John Morrissey          _o            /\         ----  __o
jwm at horde.net        _-< \_          /  \       ----  <  \,
www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__

--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au To unsubscribe, email
'majordomo at open.com.au' with 'unsubscribe radiator' in the body of the
message.

--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.