[RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch

[Hartmaier Alexander]

On 2014-03-28 09:02, Hartmaier Alexander wrote:
> On 2014-03-27 20:43, Heikki Vatiainen wrote:
>> On 03/27/2014 05:22 PM, Hartmaier Alexander wrote:
>>
>>> Did you have time to work on this feature?
>> We have worked on EAP-SIM, Diameter and other RADIUS functionality, but
>> not this. It's still on the ideas to explore list, though.
>>
>>> I've started tring to get all Radiator logs into Elasticsearch via
>>> RabbitMQ and found out that <Log FILE> also closes the pipe in case of
>>> Filename |/etc/radiator/log_to_rabbitmq.pl which is a performance problem.
>> It makes the log files easy to rotate, but I can see why it's not good
>> in this case.
> Because the logging script has some quite a 'high' startup time because
> it has to connect to RabbitMQ and also disconnects immediatly afterwards
> causing quite some overhead and RabbitMQ logging.
>>> The comment in Radius::Util::append says:
>>> # Current implementation opens, writes and closes
>>> # Future implementations might hold the file open, and reopen on
>>> # signal, or perhaps pipe to a daemon
>>>
>>> I need a feature that keeps the spawned (forked?) process open and pipe
>>> to it.
>>> I'd also need to escape the log message so my JSON doesn't break in case
>>> the message contains one or more ".
>>>
>>> LogFormat { "timestamp":"%Y-%m-%d %H:%M:%S", "priority":"%1",
>>> "message":"%2" }
>> Hmm, maybe a new class that could be configured to support the above? It
>> would need to be able to quote the log messages, maybe a configurable
>> method for different kinds of destinations (JSON, etc), hold the fd open
>> while supporting FarmSize and possibly something else too. I'd say
>> extending Log FILE may not be a good idea but to have a new logging class.
>>
>> If you already have something that does what you require on the <Log
>> ...> side, please get back to me directly.
> Yeah, I was thinking about writing a LogMessagePassing or Log4perl class
> but am not sure to which api I have to follow as its internal and as far
> as I know undocumented.
> For now I've created a named pipe which I configured Radiator to log to
> and have a Message::Passing DSL script that runs as daemon using
> Daemon::Control which uses the TailFile input and the AMQP output.
> I wanted to not have to run that extra daemon because it's one more
> thing that can go wrong.
I need a solution that outputs structed log messages serialized as JSON
fast.
My quick solution is:

<Log FILE>
   # this is a named pipe created using mkfifo
    Filename %L/radiator.fifo
    LogFormat { "timestamp":"%Y-%m-%dT%H:%M:%SZ", "source_host":"%h",
"priority":"%1", "type":"radiator",
"customer":"%{OSC-Customer-Identifier}", "message":"%2" }
    Trace 3
</Log>

The problem is that some messages contain linefeeds (openssl library
errors when using EAP-TLS for example) which don't get escaped as per
JSON spec and make the JSON parsing fail.

I think extending LogFormat is the right way to go because one might
want to log to a file or database in json or yaml as well.
What I still haven't figured out is a config format.
Enabling to pass a Perl sub to LogFormat would be the most flexible
option but not the most user friendly.

Ideas?

>
>> Thanks,
>> Heikki
>>
>>
>
>
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
> Handelsgericht Wien, FN 79340b
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> Notice: This e-mail contains information that is confidential and may be privileged.
> If you are not the intended recipient, please notify the sender and then
> delete this e-mail immediately.
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> _______________________________________________
> radiator mailing list
> radiator at open.com.au
> http://www.open.com.au/mailman/listinfo/radiator