(RADIATOR) Patch for LDAP_OPERATIONS_ERROR with half-closed TCP

[Bjoern A. Zeeb]

Hi,

if running Radiator with LDAP backends and HoldServerConnection and
those sessions are idle for too long or longer than an Idle Timeout
on the LDAP server it might happen that the LDAP server closes the
connection.
What you end up with is a half-closed TCP connection but perl-ldap
hasn't yet read the EOF with asn_read/Convert::ASN1, nor would it
really recognize this condition as a close.
Other scenarios how you can get into te half-closed/closed is with
firewalls in between that expire states and one way or the other start
telling your socket that TCP is being finished.

What happens with Radaitor in this case:

You have an supposedly alive LDAP connection.  findUser() does the
reconnect tests, getpeername() still returns soemthing valid (at least
on OSes with sane stacks;) thus reconnect returns and a search() is
started.
Now perl-ldap tries to send data, the LDAP server sends back a RST
perl-ldap tries to read the answer which doesn't make sense and
returns an LDAP_OPERATIONS_ERROR which you will find in your logs.
Radiator will think the LDAP Server is down and go into backoff mode.
Now if you have multiple servers and this happens with all of them
you are lost.

Actually the LDAP server would answer queries fine with a new
connection.

So what the attached patch does is:
if running in HoldServerConnection and we have a supposedly valid LDAP
socket, check if there is any data to read which should not be the
case in sync mode unless there is an unsolicited notification 'Notice
of Disconnection' (or an EOF pending).
If there is anything let perl-ldap process the data.
In case this returns with an LDAP_OPERATIONS_ERROR (the one serach
would have ginven us) check EVAL_ERROR which Convert::ASN1 sets in
case of an read error. If that says 'Unexpected EOF' close the TCP
sessions from our side as well.
Log the case that there was a 'Server side disconnect'.
In case of an 'Notice of Disconnection' perl-ldap will have clsoed the
connection already.

In both cases the LDAP descriptor will no longer be valid and we will
do a reconnect and a following search() would succeed and not mark the
server down.


Regards,
Bjoern

-- 
Dipl. Ing. (BA) Bjoern A. Zeeb          Research & Development
CK Software GmbH                        http://www.cksoft.de/
Schwarzwaldstr. 31                      Phone: +49 7452 889 135
D-71131 Jettingen                       Fax: +49 7452 889 136
HRB245288, Amtsgericht Stuttgart        Geschaeftsfuehrer: Christian Kratzer
-------------- next part --------------
diff -upr Radiator-4.2/Radius/Ldap.pm Radiator-4.2.bz/Radius/Ldap.pm
--- Radiator-4.2/Radius/Ldap.pm	Tue Dec 18 21:24:04 2007
+++ Radiator-4.2.bz/Radius/Ldap.pm	Thu Apr 17 11:17:54 2008
@@ -199,6 +199,34 @@ sub reconnect
     $self->close_connection() 
 	if ($self->{ld} && $self->{ld}->{net_ldap_socket} && !getpeername($self->{ld}->{net_ldap_socket}));
 
+    # In case we have a persistent server connection there are events that
+    # can drop this connection. Possible causes could be a firewall timeout
+    # sending a RST or ICMP Message or an LDAP Server 'Idletimeout' with or
+    # without an unsolicited notification 'Notice of Disconnection'.
+    # Unfortunately Net::LDAP does not have any sane way to handle all but
+    # the 'Notice of Disconnection' before any operation like search().
+    # All you get is an LDAP_OPERATIONS_ERROR afterwards.
+    # Work aroudn this if we have a 'valid' persistent server connection.
+    if ($self->{HoldServerConnection} && $self->{ld} && $self->{ld}->socket()) {
+      # See if there is any pending input
+      # (which should not be the case with sync operation).
+      my $rin = '';
+      vec($rin, fileno($self->{ld}->socket()), 1) = 1;
+      my $nfound = select($rin, undef, undef, 0);
+      # There is something to read but we did not expect anything. EOL?
+      if ($nfound) {
+        # Let Net::LDAP read the incoming data (message).
+        my $code = $self->{ld}->process();
+        if ($code == Net::LDAP::Constant->LDAP_OPERATIONS_ERROR && $@ eq 'Unexpected EOF') {
+          $self->log($main::LOG_INFO, "Server side disconnect (server $self->{Host}:$self->{Port}).");
+          # Cleanly shutdown the socket using Net::LDAP function.
+          $self->{ld}->_drop_conn($self->{ld}, Net::LDAP::Constant->LDAP_SERVER_DOWN, "Server side disconnect");
+          # Also clear our LDAP handle.
+          $self->close_connection();
+        }
+      }
+    }
+
     return 1 if $self->{ld}; # We are already connected
     return 0 if time < $self->{backoff_until};