(RADIATOR) Log message "Too many open files"

[Patrick, Robert]

FYI,

I've applied the latest patch set and I'm still seeing a high number
(hundreds) of persistent/active TCP sessions for TACACS.

The failed login attempts on my network are caused by (a) CiscoWorks
failed logins, or (b) vulnerability scanning with brute-force login
attempts enabled.

These failed login attempts result in a large number of the older Cisco
devices on my network (Catalyst 2948G, 4000, 5500, etc.) holding
multiple, persistent TACACS connections open with the Radiator server.
These sessions do not appear to time-out; I have to restart the service
to close the connections.  Radiator server platform is CentOS Linux
(equal to Red Hat Enterprise Linux) version 4, with all of the latest
updates (4.3) applied to a minimum install with only a few extra
packages installed (e.g. Radiator, Nagios, Smokeping).

After disabling CiscoWorks login attempts, and not conducting any
vulnerability scans, the older Cisco devices aren't a problem.

Is there any change to the config file needed to enable TACACS timeouts
to drop these stale connections?



-----Original Message-----
From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au] On
Behalf Of Patrick, Robert
Sent: Sunday, April 16, 2006 1:34 PM
To: Mike McCauley
Cc: radius email list
Subject: RE: (RADIATOR) Log message "Too many open files"

Thanks for the quick response!

I've downloaded the updated patch set and will give it a try this
evening.

Any chance an updated RPM can be released soon? 


-Rob Patrick


-----Original Message-----
From: Mike McCauley [mailto:mikem at open.com.au]
Sent: Sunday, April 16, 2006 2:26 AM
To: Patrick, Robert
Cc: radius email list
Subject: Re: (RADIATOR) Log message "Too many open files"

Hello Robert, 

Thanks for check the facts on this problem.
We have now added a workaround for this, so that the TCP session is
closed by Radiator after a TACACSPLUS authentication failure.
The fix is in the latest Radiator patch set.
Hope that helps.

Cheers.

On Sunday 16 April 2006 14:12, Patrick, Robert wrote:
> I have confirmed that many of the older Cisco switches on my network 
> (we have hundreds of Cisco switches, many still run older IOS & CatOS
> versions) don't close their TCP sessions during login failures via 
> TACACS.  This results in Radiator having many open sessions when
viewed
> by netstat.   Restarting Radiator causes the sessions to quickly
timeout
> and drop off.
>
> What can I do so Radiator is immune to this "bug" in the older Cisco 
> devices?  Is there a timeout value that can be set?
>
> As a workaround I'm considering restarting Radiator every hour via 
> cron...kludge.
>
> -----Original Message-----
> From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au]
> On Behalf Of Mike McCauley
> Sent: Sunday, February 26, 2006 4:39 AM
> To: Patrick, Robert
> Cc: radius email list
> Subject: Re: (RADIATOR) Log message "Too many open files"
>
> Hello Robert,
>
> Does this mean that all those TACACS authentication sessions are still

> in progress, or are they completed, but the TCP connection is still in

> place? ie what does netstat report for all those telnet client 
> connections?
>
> Is it possible the TELNET client in your routers do not close the TCP 
> connection properly/at all after authentication?
>
> Is there some way you can distinguish between the scanning attempts 
> and legitimate login attempts?
>
> You dont mention what operating system you are using, but most 
> operating systems enforce limits on the number of simultaneously open 
> files for a single process. And most allow you to change that limit.
> So, if you can be sure that you can increase the open file limit until

> it is above the maximum number of simultaneous telnet sessions, you
should do that.
>
> I dont think this is a bug in Radiator, but if you could send me a
> (sanitized) excerpt from your Radiator log file showing what happens 
> at the end of one of these bogus sessions, it would help me to decide.

> I am particularly interested if you see a TacacsplusConnection 
> disconnected from ....
> line for each connection.
>
> Cheers.
>
> On Sunday 26 February 2006 04:53, Patrick, Robert wrote:
> > Seeking so find a way for Radiator to withstand brute force login 
> > attempts...
> >
> > During periodic network vulnerability scanning all of our switches 
> > and
> >
> > routers get hit with a ton of a telnet brute-force login attempts.
> > These are all sent via TACACS to Radiator.  Soon after the scans 
> > start, I'm seeing the below error messages in 
> > /var/log/radius/logfile,
> >
> > and it doesn't seem to clear until I restart the process.
> >
> > What can I do so that Radiator avoids this failure, while still 
> > allowing the brute force attempts to be denied, meanwhile allowing 
> > any
> >
> > valid logins?  TACACS logins are checked against a flat file. lsof 
> > showed 4251 lines, 1008 of which were TACACS connections.  Netstat 
> > output showed 447 TACACS connections, out of 527 total lines.
> >
> > Sat Feb 25 13:06:39 2006: ERR: Could not accept on Tacacs listen
>
> socket:
> > Too many open files
> > Sat Feb 25 13:06:41 2006: ERR: Could not accept on Tacacs listen
>
> socket:
> > Too many open files
> >
> >
> > Thanks,
> >
> > -Rob Patrick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.open.com.au/pipermail/radiator/attachments/20060522/552591a8/attachment.html>