[RADIATOR] Hello guys - losing some "calls" when I get some peaks (per second)

[Christian Kratzer]

Hi Ricardo,

On Mon, 8 Nov 2010, Ricardo Freitas wrote:

> Hey guys
>
> I have been using Radioator for the past years (from 2006) and it is
> mainly used to get calls, validating them and then saving them in a
> database.
>
> The whole system has been working smoothly until now, where we have more
> calls than ever.
>
> The main problem isn't with the number of calls in a whole, but in the
> time frame they are received. From what I could learn, I think we manage
> to successfully account for, more or less, 130 calls in a certain second
> where the rest of the incoming calls are either lost or, and this is
> where my doubt is, saved in a temporary cache.
>
> I don't know where or how to configure this but I would like to know if
> it is possible to increase this cache size or, at least, know how is it
> configure.
>
> Let me refrain that ALL calls appear on the radius server in the log
> files with all their specific info (msisdn, alias, date of call, etc)
> but not on the database, somehow they are not getting sent to the database.
>
> Let me just explain with an example.
>
> Imagine a certain second or seconds where you have 100, 60, 70 and 50
> calls on four consecutive seconds. All calls are saved in the database
> (well, you can lose one or two).
>
> Now lets step it up a notch. Imagine you have 4 consecutive seconds with
> 130, 180, 200, 170 calls. This time, and despite ALL the calls are saved
> in the radius log servers, some of them are not passed through the mysql
> database server, to be more specific in this case we generally lose 1/3
> or so of the calls (higher the number, higher the loss - we guess the
> "safe" mark is 130 - above that all hell breaks loose).
>
> Do you guys have any idea how to solve this?
>
> Currently configuration - at least what I can tell you know - we have 6
> routers connected to our radius server, with four different instances on
> the server - my guess, this was to distribute the load of the routers in
> a more share way.
>
> First things first: is there a place where I can check the so called
> cache? I mean, when we have lower load seconds, the database sometimes
> has MORE calls that the radius itself which makes me believe it's
> somehow dumping the cached calls..
>
> Thanks a lot, this is getting very problematic.

sounds like your database is not managing to keep up although it's hard to say exactly where your calls are getting lost from your description.

You say all calls reach your radius server.  From what logs are you determining this ? Might be interesting to know how far they get in.  It is possible your database inserts are taking too long and the queued requests are getting dropped.

If I understand you correctly you just need to save off accounting to a database without any authentication taking place.  An extremely scalable strategy for this kind of situation might be to decouple the receiving of the requests and the insertion in the datbase.

That is setup radiator to write the accounting to a logfile in the filesystem and setup a batched process to process completed logfiles every couple of minutes.  Radiator will have no trouble writing 200 calls/sec into an ascii line by line log.  I would guess the limit will be even higher.  The separete process can insert the records into the database withough danger of timeouts occurring.


Greetings
Christian

-- 
Christian Kratzer                      CK Software GmbH
Email:   ck at cksoft.de                  Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0          D-71126 Gaeufelden
Fax:     +49 7032 893 997 - 9          HRB 245288, Amtsgericht Stuttgart
Web:     http://www.cksoft.de/         Geschaeftsfuehrer: Christian Kratzer