[RADIATOR] Memory Leak on RHEL 8.5

Wolfgang Breyha radiator at blafasel.at
Mon Apr 4 18:14:41 UTC 2022


Hi!

We (University of Vienna) recently noticed performance issues with our new
Radiator Servers running on RHEL 8.5. These were caused by radiusd (4.26)
itself eating memory until the machines started swaping and the IO-waits
raised until radiusd wasn't able to handle the load anymore.

The same config was running on RHEL 6 with radiator 4.18 without any
problems or noticeable leaks for years.

I tried to run radiusd under valgrind with our configuration and assuming
that perl itself is not the cause I think the leak(s) happen(s) somewhere
down the Net::SSLeay->opennssl pipe. Since our config is rather complex I
tried to set up a simple EAP example config to see if it happens there as
well... and "luckily" it does. config is attached.

I then start eapol_test (from wpa_supplicant RPM) with a config of
network={
eap=PEAP
eapol_flags=0
key_mgmt=IEEE8021X
identity="testuser"
anonymous_identity="anonymous"
password="testpass"
ca_cert="/etc/pki/tls/cert.pem"
phase2="auth=MSCHAPV2"
}
in a loop and can watch radiusd eating memory.

I started it in batches of 1000 and the RSS increased from fresh start...
36884->115592->124648->132256->135120->139868->139868->143116->147152

If I run this config with valgrind again I find a similar "definitely lost
memory" section with a close amount of requests as with our full config:
==1420461== 233,728 bytes in 913 blocks are definitely lost in loss record
6,062 of 6,088
==1420461==    at 0x4C360A5: malloc (vg_replace_malloc.c:380)
==1420461==    by 0xA39690C: CRYPTO_zalloc (in /usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0xA382AC3: EVP_PKEY_meth_new (in
/usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0xCF3CAD7: ??? (in /usr/lib64/engines-1.1/pkcs11.so)
==1420461==    by 0xA3648E4: ENGINE_get_pkey_meth (in
/usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0xA382EA4: ??? (in /usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0xA37E543: ??? (in /usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0x9FD5A41: ??? (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FC833E: ??? (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FB3C97: SSL_do_handshake (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9D3ACA3: ??? (in
/usr/lib64/perl5/vendor_perl/auto/Net/SSLeay/SSLeay.so)
==1420461==    by 0x4F2F4B8: Perl_pp_entersub (in /usr/lib64/libperl.so.5.26.3)

and a "possibly lost memory"
==1420461== 640,000 bytes in 1,000 blocks are possibly lost in loss record
6,079 of 6,088
==1420461==    at 0x4C360A5: malloc (vg_replace_malloc.c:380)
==1420461==    by 0xA39690C: CRYPTO_zalloc (in /usr/lib64/libcrypto.so.1.1.1k)
==1420461==    by 0x9FBA5AC: SSL_SESSION_new (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FBAE06: ??? (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FD9E78: ??? (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FC855A: ??? (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9FB3C97: SSL_do_handshake (in /usr/lib64/libssl.so.1.1.1k)
==1420461==    by 0x9D3ACA3: ??? (in
/usr/lib64/perl5/vendor_perl/auto/Net/SSLeay/SSLeay.so)
==1420461==    by 0x4F2F4B8: Perl_pp_entersub (in /usr/lib64/libperl.so.5.26.3)
==1420461==    by 0x4F27324: Perl_runops_standard (in
/usr/lib64/libperl.so.5.26.3)
==1420461==    by 0x4EA6FFC: perl_run (in /usr/lib64/libperl.so.5.26.3)
==1420461==    by 0x108ED9: ??? (in /usr/bin/perl)

The machines run a fully patched RHEL 8.5 with the current
radiator-4.26-1.el8.noarch from your website
openssl-libs-1.1.1k-5.el8_5.x86_64
perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+594cad75.x86_64

I also tried to build a new Net::SSLeay-1.92. Same results.

If we can't find the cause it seems we need to restart radiator periodically.

With kind regards,
Wolfgang Breyha
-- 
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria
-------------- next part --------------
# eap_tls.cfg
#
# Example Radiator configuration file.
# This very simple file will allow you to get started with EAP TLS
# authentication. We suggest you start simple, prove to yourself that
# it works and then develop a more complicated configuration.
#
# This example will authenticate from a standard users file in the
# current directory. It will accept requests from any client and try
# to handle request for any realm. And it will print out what its
# doing in great detail.
#
# In order to authenticate, the clients user name must be in %D/users
# (the password is irrelevant for EAP TLS).
# It will also require that the certificate installed on the client
# is within one step of the root certificate, and that the subject name
# in the client certificate is the same as the user name they are trying
# to log in as.
#
# In order to test this, you can use the sample test certificates
# supplied with Radiator. For production, you WILL need to install a
# real valid server certificate and key for Radiator to use.
#
# See radius.cfg for more complete examples of features and syntax,
# and refer to the reference manual for a complete description of all
# the features and syntax.
#
# You should consider this file to be a starting point only
# $Id$

# Use a lower logging trace level in production systems.
Trace 4

# Add request identifier and high precision timestamp to log messages.
LogTraceId
LogMicroseconds

# DbDir sets the value of %D. LogDir sets the value of %L. Additional
# configuration files go to DbDir. LogDir typically contains Radiator
# log file and optionally authentication and accounting logs.
DbDir   /etc/radiator
LogDir  /var/log/radiator
LogFile %L/radius.log

# Certificates and related files are in %{GlobalVar:CertDir}, see below.
DefineGlobalVar CertDir /opt/radiator/radiator/certificates

# Any custom dictionaries should go in DbDir.
DictionaryFile /opt/radiator/radiator/dictionary

# AuthPort and AcctPort default to 1645 and 1646.
AuthPort 1645,1812
AcctPort 1646,1813


# You will probably want to add other Clients to suit your site,
# one for each NAS you want to work with
<Client DEFAULT>
	Secret	mysecret
</Client>

<AuthBy GROUP>
	Identifier AuthTEST
	NoEAP
	<AuthBy FILE>
		Filename /root/eapoltests/users
		EAPType PEAP,LEAP,TTLS,MSCHAP-V2
		EAPTLS_CAFile /etc/pki/tls/certs/xxxxxx.crt
		EAPTLS_CertificateFile /etc/radiator/ssl/xxxxxxxxxxxxxxxxxxx.crt
		EAPTLS_CertificateChainFile /etc/radiator/ssl/xxxxxxxxxxxxxxxxxxx.crt
		EAPTLS_CertificateType PEM
		EAPTLS_PrivateKeyFile /etc/radiator/ssl/xxxxxxxxxxxxxxxxxxx.key
		EAPTLS_MaxFragmentSize 1000
		EAPTLS_NoCheckId
		AutoMPPEKeys
	</AuthBy>
</AuthBy>


<Handler TunnelledByPEAP=1>
	RejectHasReason
	AuthByPolicy ContinueWhileAccept
	AuthBy AuthTEST
</Handler>


<Handler TunnelledByTTLS=1>
	RejectHasReason
	AuthByPolicy ContinueWhileAccept
	AuthBy AuthTEST
</Handler>


<Handler>
	<AuthBy GROUP>
		NoEAP
		AuthByPolicy ContinueWhileAccept
		AuthBy AuthTEST
	</AuthBy>
</Handler>



More information about the radiator mailing list