(RADIATOR) Threading support (again)
Hugh Irvine
hugh at open.com.au
Mon Sep 3 18:29:35 CDT 2007
Hello Martin -
Thanks very much for your considered comments.
I concur with your analysis that Radiator is mostly the "meat in the
sandwich" with respect to the RADIUS requests being generated by the
NAS equipment on the frontend and the SQL database on the backend. It
is essential when designing such a system that the overall
characteristics be considered carefully. Clearly there is no point in
configuring a NAS to generate accounting alives too frequently (there
is no hard and fast rule, but I can't imagine a situation where you
would need more than one per hour at most). Equally there is no point
in creating huge accounting tables in the backend SQL database - a
much better approach is to use a different accounting table (easily
accomplished using "special" characters for the table name) for each
day and archive the completed ones during "quiet" times. If the data
is required for billing and customer access it can be postprocessed
on a separate database server on a daily basis.
The RADONLINE table is always a problem in high-volume systems, but
it can be impoved by using stored procedures and/or functions (recent
patches to Radiator 3.17.1 also support bind variables for those
databases that support them). The RADONLINE table can also be run in
a RAM disk if you have the appropriate OS and database support.
You can run multiple connections to the database simply by specifying
multiple DBSource lines (ie. one for authentication, one for
accounting, one for RADONLINE, etc). However, you need to make
absolutely certain that this is really going to improve things (YMMV).
Another big problem that is commonly seen is the flood of access
requests that occurs when a NAS concentrator is bounced - such a
device terminating several thousand DSL connections will swamp any
RADIUS backend system. If your NAS equipment supports rate-limiting,
you can configure it there (preferred), alternatively I have written
a couple of different versions of rate-limiting hooks that are
included in the Radiator distribution in "goodies/hooks.txt".
As has been mentioned before, it is always a good idea to run two
instances of Radiator - one for authentication and the other for
accounting. This is easy to do with a singe configuration file
together with parameters passed in on the command line at startup. In
my consulting practice I often end up designing systems with a
"frontend" instance of Radiator which is used to classify the RADIUS
requests into "services" and proxy them to multiple "backends", such
as "dialup", "DSL", "VPN", "Wireless", etc. on the same host. This
makes much better use of multi-processor machines (think of it as
controlled multi-threading), and it also has the added benefit of
making each separate configuration file much simpler.
Most large-scale RADIUS systems run two hardware load-balancers
(primary and secondary RADIUS targets as seen from the NAS
equipment), in front of multiple Radiator hosts configured as above,
with a very fast high-availablity SQL database host on the backend.
Many thanks for a very useful discussion - if anyone has additional
comments I would be most interested to see them.
BTW - we are available on a contract basis for design, implementation
and training as required.
best regards
Hugh
On 4 Sep 2007, at 02:05, Martin Wallner wrote:
> Hi!
>
> Same problems here (peaks and so on)... where I'm not thinking that -
> with current hardware - the problem does not lie in the amount of
> requests and records that are sent to RADIATOR, the problem lies in
> the
> authentication and accounting backbone of the system.
>
> Accounting inserts sometimes (and especially with excessive need of
> re-writing the indicies) are a terrible strain on the DB, which then
> goes back to Radiator, which is waiting for the ack to the db, that
> sometime goes in to timeout, and so on.
>
> What's also straining is the amount of Alives, that starts to mount up
> when you are getting more and more users.
> The only valid and usable method to store the Accounting Infos is
> in SQL
> (sooner or later it goes in some form of DB anyway, so why handling
> and
> transforming GIGANTIC text files in the DB when one can put it in
> the db
> directly)
>
> Currently in the 'goodies', the SQL preposition is to write the
> SQL-Accounting table like a text file, just slop the Accounting
> Data in
> the table and be done with it (which makes for a LOT of WASTE in the
> Accounting DB, especially, because normally your accounting dept. is
> interested in either the last Alive record or the Stop record). This
> makes f.e. our current setup write up to 1 Mil+ entries in the DB
> within
> 24 hours.... I am NOT criticising the way it's done in 'goodies' but
> it's for low to medium high usage. The database server can also
> start to
> 'split' up the data to make something like getting data for usage for
> the customer easier and so on... Basically, that means, that per
> day and
> session a maximum of 3 entries (Start - 1 time, Alive - n times
> updated,
> and Stop - 1 time)... also, it would not need the Index to become
> monsterous... (we are just testing this here and we get around 25% of
> the Records instead of the 1 Mil +....) and you can do nice
> preprocessing in the SQL Server of the data... Maybe I can get my
> bosses
> to put this into RADIATOR-pd...
>
> Another way - maybe - is to make more than one connections to the
> Database; f.e. make more SQL-Authentication Clauses for the different
> Realms or handlers, bringing Radiator to open up more DB-Connections
> (never tested this...)
>
> I'm not thinking that asynchronous DBI will help here much, because
> the
> strain is still here, and, even worse, Radiator wouldn't be able to
> clean out the complete transaction, he would have to store it until it
> gots acked or Radiator calls for the ack...
>
> Additionally I found out that the current way of doing the
> radonline-table is also making up a lot of clutter in the index and
> (I'm
> using PostGreSQL) running the vacuum daemon is mandatory...
>
> So, in conclusion, besides from running up to 4 Radiators (2
> Accounting
> / 2 Authenticatings) on a 4 Core Server, doing LDAP Authentication and
> SQL Accounting, we should first and furthermore see what we can do at
> the Interfaces and the Handling of the queues to the backbone services
> and get this more streamlined. The current Radiator Core is (on a 4
> Core
> HP ProLiant 360 G4p ) good for 200-300 Transactions per second (all
> local userdb and local accounting file).... The moment one tries to
> make
> the work easier by bringing in more sophisticated stuff, it can get
> ugly...
>
> Martin Wallner
> IP/Systems sen. Engineer
> eTel Austria AG, Haydngasse 17, A-1060 Wien
> Tel.: +43 (0)50 1011 - 2254
> Mobil: +43 (0)699 1599 - 2254
> E-Mail: martin.wallner at etel.at
> www.etel.at
>
>
> -----Original Message-----
> From: owner-radiator at open.com.au [mailto:owner-
> radiator at open.com.au] On
> Behalf Of Hartmaier Alexander
> Sent: Montag, 03. September 2007 15:55
> To: radiator at open.com.au
> Subject: RE: (RADIATOR) Threading support (again)
>
> Hi!
>
> We encountered similar problems in the last few month when traffic
> rises.
> Have you thought about using POE and various components (like
> asynchronous DBI
> calls) for Radiator?
> For us it seems that the accounting inserts block radiator sometimes.
> How could be snmp checks be logged to verify that aren't the cause of
> our problems?
> I never saw a single log line for those even with a trace 5 log.
>
> With best regards
> Alexander Hartmaier
>
> T-Systems Austria GesmbH
> Rennweg 97-99
> A-1030 Vienna
>
> phone: +43-(0)57057-4320
> mobile: +43-(0)676-8642-4320
> -----Original Message-----
> From: owner-radiator at open.com.au [mailto:owner-
> radiator at open.com.au] On
> Behalf Of John Morrissey
> Sent: Friday, August 24, 2007 11:46 PM
> To: radiator at open.com.au
> Subject: (RADIATOR) Threading support (again)
>
> Looking through the archives, it's been a long time since someone's
> kicked the threading horse.
>
> We've been running Radiator on a fairly substantial RADIUS
> installation
> (a dozen or so machines in three locations across the United
> States) for
> several years now, making use of an LDAP directory for account
> storage.
> (Hugh or Mike, feel free to e-mail me privately if you need more
> information.) Unfortunately, OpenLDAP libraries do not seem well
> behaved
> when Radiator forks, although it's been a while since we've tried.
>
> Our largest problem with a monolithic process is that we've had
> occasional problems where device outages cause thousands of
> subscribers
> to reauthenticate simultaneously. This overwhelms our authentication
> radiusd instances, causing UDP network queues on that port to
> skyrocket,
> eventually leading to a cascade failure that is obnoxious to recover
> from, to say the least. Our RADIUS servers are by no means
> underpowered,
> but it's just not cost-effective to maintain a huge (and otherwise
> idle)
> hardware pool just to handle occasional huge spikes like this.
>
> Additionally, we're starting to deploy machines with four CPU cores as
> their price has dropped substantially. We like sticking to a fairly
> standard configuration, so we'd like to avoid "wasting" hardware that
> Radiator will never use (we have auth and accounting split into two
> Radiator instances).
> We could run more radiusd processes on different ports or IP
> addresses,
> but this seems wasteful and complex.
>
> Lastly, we're looking at implementing simultaneous use restrictions
> and
> are concerned that the state refresh (SNMP in some cases, locally
> implemented interfaces to proprietary devices in others) will block
> radiusd excessively.
> We're also looking at proxying to more customers, and are concerned
> that
> unreachable or underperforming proxy destinations will cause the same
> behavior.
>
> We had considered implementing rate limiting to handle our first
> concern
> above, but that doesn't get us anywhere with the others. To that end,
> I'm toying with the idea of adding a new process model to Radiator
> somewhat akin to Apache's prefork MPM, with a number of child
> processes
> (not threads) running round-robin on the incoming UDP port or handling
> work directed at them via some sort of IPC from a central request
> handler process.
>
> I haven't dived into the code yet to see how non-trivial this will be,
> and I thought I'd ask around here first to see what everyone's
> thoughts
> are. I know that a threaded model hasn't been implemented due to
> alpha-quality thread support in Perl, and I haven't looked into its
> current state yet.
> That's part of the reason I'm keen on using separate worker processes
> instead of threads.
>
> thanks,
> john
> --
> John Morrissey _o /\ ---- __o
> jwm at horde.net _-< \_ / \ ---- < \,
> www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
>
> --
> Archive at http://www.open.com.au/archives/radiator/
> Announcements on radiator-announce at open.com.au To unsubscribe, email
> 'majordomo at open.com.au' with 'unsubscribe radiator' in the body of the
> message.
>
>
> --
> Archive at http://www.open.com.au/archives/radiator/
> Announcements on radiator-announce at open.com.au
> To unsubscribe, email 'majordomo at open.com.au' with
> 'unsubscribe radiator' in the body of the message.
NB:
Have you read the reference manual ("doc/ref.html")?
Have you searched the mailing list archive (www.open.com.au/archives/
radiator)?
Have you had a quick look on Google (www.google.com)?
Have you included a copy of your configuration file (no secrets),
together with a trace 4 debug showing what is happening?
Have you checked the RadiusExpert wiki:
http://www.open.com.au/wiki/index.php/Main_Page
--
Radiator: the most portable, flexible and configurable RADIUS server
anywhere. Available on *NIX, *BSD, Windows, MacOS X.
Includes support for reliable RADIUS transport (RadSec),
and DIAMETER translation agent.
-
Nets: internetwork inventory and management - graphical, extensible,
flexible with hardware, software, platform and database independence.
-
CATool: Private Certificate Authority for Unix and Unix-like systems.
--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.
More information about the radiator
mailing list