(RADIATOR) Log message "Too many open files"

[Mike McCauley]

Hello Robert,

On Tuesday 23 May 2006 13:00, Patrick, Robert wrote:
> FYI,
>
> I've applied the latest patch set and I'm still seeing a high number
> (hundreds) of persistent/active TCP sessions for TACACS.
>
> The failed login attempts on my network are caused by (a) CiscoWorks
> failed logins, or (b) vulnerability scanning with brute-force login
> attempts enabled.
>
> These failed login attempts result in a large number of the older Cisco
> devices on my network (Catalyst 2948G, 4000, 5500, etc.) holding
> multiple, persistent TACACS connections open with the Radiator server.
> These sessions do not appear to time-out; I have to restart the service
> to close the connections.  Radiator server platform is CentOS Linux
> (equal to Red Hat Enterprise Linux) version 4, with all of the latest
> updates (4.3) applied to a minimum install with only a few extra
> packages installed (e.g. Radiator, Nagios, Smokeping).
>
> After disabling CiscoWorks login attempts, and not conducting any
> vulnerability scans, the older Cisco devices aren't a problem.
>
> Is there any change to the config file needed to enable TACACS timeouts
> to drop these stale connections?

No config change required. You should see the tacacs connection disconnected 
when the authentication fails. Perhaps you can send me some of your log file 
showing what happens during some of these CiscoWorks
failed logins, or (b) vulnerability scanning with brute-force login
attempts enabled. The I can confirm if the patch is enabled for you.

Cheers.

>
>
>
> -----Original Message-----
> From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au] On
> Behalf Of Patrick, Robert
> Sent: Sunday, April 16, 2006 1:34 PM
> To: Mike McCauley
> Cc: radius email list
> Subject: RE: (RADIATOR) Log message "Too many open files"
>
> Thanks for the quick response!
>
> I've downloaded the updated patch set and will give it a try this
> evening.
>
> Any chance an updated RPM can be released soon?
>
>
> -Rob Patrick
>
>
> -----Original Message-----
> From: Mike McCauley [mailto:mikem at open.com.au]
> Sent: Sunday, April 16, 2006 2:26 AM
> To: Patrick, Robert
> Cc: radius email list
> Subject: Re: (RADIATOR) Log message "Too many open files"
>
> Hello Robert,
>
> Thanks for check the facts on this problem.
> We have now added a workaround for this, so that the TCP session is
> closed by Radiator after a TACACSPLUS authentication failure.
> The fix is in the latest Radiator patch set.
> Hope that helps.
>
> Cheers.
>
> On Sunday 16 April 2006 14:12, Patrick, Robert wrote:
> > I have confirmed that many of the older Cisco switches on my network
> > (we have hundreds of Cisco switches, many still run older IOS & CatOS
> > versions) don't close their TCP sessions during login failures via
> > TACACS.  This results in Radiator having many open sessions when
>
> viewed
>
> > by netstat.   Restarting Radiator causes the sessions to quickly
>
> timeout
>
> > and drop off.
> >
> > What can I do so Radiator is immune to this "bug" in the older Cisco
> > devices?  Is there a timeout value that can be set?
> >
> > As a workaround I'm considering restarting Radiator every hour via
> > cron...kludge.
> >
> > -----Original Message-----
> > From: owner-radiator at open.com.au [mailto:owner-radiator at open.com.au]
> > On Behalf Of Mike McCauley
> > Sent: Sunday, February 26, 2006 4:39 AM
> > To: Patrick, Robert
> > Cc: radius email list
> > Subject: Re: (RADIATOR) Log message "Too many open files"
> >
> > Hello Robert,
> >
> > Does this mean that all those TACACS authentication sessions are still
> >
> > in progress, or are they completed, but the TCP connection is still in
> >
> > place? ie what does netstat report for all those telnet client
> > connections?
> >
> > Is it possible the TELNET client in your routers do not close the TCP
> > connection properly/at all after authentication?
> >
> > Is there some way you can distinguish between the scanning attempts
> > and legitimate login attempts?
> >
> > You dont mention what operating system you are using, but most
> > operating systems enforce limits on the number of simultaneously open
> > files for a single process. And most allow you to change that limit.
> > So, if you can be sure that you can increase the open file limit until
> >
> > it is above the maximum number of simultaneous telnet sessions, you
>
> should do that.
>
> > I dont think this is a bug in Radiator, but if you could send me a
> > (sanitized) excerpt from your Radiator log file showing what happens
> > at the end of one of these bogus sessions, it would help me to decide.
> >
> > I am particularly interested if you see a TacacsplusConnection
> > disconnected from ....
> > line for each connection.
> >
> > Cheers.
> >
> > On Sunday 26 February 2006 04:53, Patrick, Robert wrote:
> > > Seeking so find a way for Radiator to withstand brute force login
> > > attempts...
> > >
> > > During periodic network vulnerability scanning all of our switches
> > > and
> > >
> > > routers get hit with a ton of a telnet brute-force login attempts.
> > > These are all sent via TACACS to Radiator.  Soon after the scans
> > > start, I'm seeing the below error messages in
> > > /var/log/radius/logfile,
> > >
> > > and it doesn't seem to clear until I restart the process.
> > >
> > > What can I do so that Radiator avoids this failure, while still
> > > allowing the brute force attempts to be denied, meanwhile allowing
> > > any
> > >
> > > valid logins?  TACACS logins are checked against a flat file. lsof
> > > showed 4251 lines, 1008 of which were TACACS connections.  Netstat
> > > output showed 447 TACACS connections, out of 527 total lines.
> > >
> > > Sat Feb 25 13:06:39 2006: ERR: Could not accept on Tacacs listen
> >
> > socket:
> > > Too many open files
> > > Sat Feb 25 13:06:41 2006: ERR: Could not accept on Tacacs listen
> >
> > socket:
> > > Too many open files
> > >
> > >
> > > Thanks,
> > >
> > > -Rob Patrick

-- 
Mike McCauley                               mikem at open.com.au
Open System Consultants Pty. Ltd            Unix, Perl, Motif, C++, WWW
9 Bulbul Place Currumbin Waters QLD 4223 Australia   http://www.open.com.au
Phone +61 7 5598-7474                       Fax   +61 7 5598-7070

Radiator: the most portable, flexible and configurable RADIUS server 
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald, 
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS, 
TTLS, PEAP etc on Unix, Windows, MacOS, NetWare etc.

--
Archive at http://www.open.com.au/archives/radiator/
Announcements on radiator-announce at open.com.au
To unsubscribe, email 'majordomo at open.com.au' with
'unsubscribe radiator' in the body of the message.