[RADIATOR] Memory Leak on RHEL 8.5

[Wolfgang Breyha]

Hi!

A question left from your previous answer:
Which version does el9 beta use? I see they switched to OpenSSL 3 already
searching the net, right?

On 06/04/2022 18:27, Heikki Vatiainen wrote:
> Unfortunately it seems not to be that easy. I used
> openssl-1.1.1k-5.el8_5.src.rpm and tried to compile a local copy. After
> applying the patches and compiling with parameters from openssl.spec, I was
> able to create a version that works.
> 
> Problem is that it works too well. It does not leak. Compiler flags, see
> openssl version -a, match the system version and everything looks equal
> (except of built date, etc. changes that are expected).

Really? That's interesting, because meanwhile I tried the following things:
*) rebuild 1.1.1k from el8 src RPM
*) rebuild the current openssl-1.1.1n-1.fc35.src.rpm for el8
*) remove several patches from the 1.1.1n spec in steps to the point of
nearly destroying the host ending up with broken ssh/dnf due to missing
krb5_evp symbols. Luckily wget still worked with https to fetch the
original RPMS;-)

But even then ... no change in memory leaks for radiusd.

*) tried to build the rpm with all patches but with the "unhobbled" full
openssl-1.1.1n tarball.

No difference as well. It leaks.

I also tried another tool for memleak detection called memleax ...
https://github.com/WuBingzheng/memleax
... which is able to attach to a running PID. This detects the same leak as
well:
CallStack[24]: may-leak=6 (1536 bytes)
    expired=6 (1536 bytes), free_expired=0 (0 bytes)
    alloc=33 (8448 bytes), free=0 (0 bytes)
    freed memory live time: min=0 max=0 average=0
    un-freed memory live time: max=11
    0x00007fbbc4301a90  libc-2.28.so  __malloc()+0
    0x00007fbbc211b90d  libcrypto.so  CRYPTO_zalloc()+13
    0x00007fbbc2107ac4  libcrypto.so  EVP_PKEY_meth_new()+36
    0x00007fbbc154ead8  pkcs11.so
    0x00007fbbc20e98e5  libcrypto.so  ENGINE_get_pkey_meth()+53
    0x00007fbbc2107ea5  libcrypto.so
    0x00007fbbc2103544  libcrypto.so
    0x00007fbbc24d7a42  libssl.so
    0x00007fbbc24ca33f  libssl.so
    0x00007fbbc24b5c98  libssl.so  SSL_do_handshake()+88
    0x00007fbbc2767e84  SSLeay.so
    0x00007fbbc55144b9  libperl.so  Perl_pp_entersub()+505
    0x00007fbbc550c325  libperl.so  Perl_runops_standard()+53
    0x00007fbbc548bffd  libperl.so  perl_run()+797
    0x0000562b6f5f7eda  perl
    0x00007fbbc429f493  libc-2.28.so  __libc_start_main()+243
    0x0000562b6f5f7f1e  perl


Then I used my locally built openssl 1.1.1n RPMs I use on other el8 hosts.
These use a clean openssl source without RH patches, slightly different
Configure and builds to /usr/local/openssl. I then rebuilt the Net::SSLeay
1.92 RPM I already built before with OPENSSL_PREFIX to use my custom OpenSSL.

This way I can easily switch between original and custom OpenSSL by
downgrading/updating Net::SSLeay.

And with the custom OpenSSL RPMs it seems radiusd doesn't leak. At least
valgrind/memleax do not detect the SSL_do_handshake leak and radiusd memory
footprint gets stable after the first batch of requests floating at
~110-120MB. I put this setup on one of our production hosts now to see what
happens as soon as people start connecting their WLAN devices in the morning.

> It might be that it's not a problem caused by patches the source rpm
> contains, but something else. The build I did was manual done by calling
> ./Configure ... after patching the OpenSSL source.

Hmmm, I can't confirm this since all the RPMs I used in my tests are built
on Rocky 8.5 as well. The Fedora 35 SPEC doesn't differ in any important
aspect.

I rebuilt 1.1.1k with the RHEL8 spec. So, this should be the exact same
result as your local build except the rpmbuild environment (which sets a
lot of compiler/linker defaults which you compared).

But only the version without the RH hobble/fips/patches-stuff works which
uses the same rpmbuild env and the SPEC file I use is a modification of the
original one.

So my guess for now is that the issue is caused by the patches (but not the
hobbled part). Since wide parts of the patches depend on each other I see
no easy way to pin it to one of them.

Greetings, Wolfgang