[RADIATOR] memory leaks

[Heikki Vatiainen]

On 29/11/2019 11.57, Jan Tomasek wrote:

> after I started using FarmSize = 3 I've noted that Radiator is quite
> badly leaking memory, see:
>      http://tomasek.cz/stuff/memusage_radiusd-day.png
> 
> I'm not sure that the problem comes with FarmSize. The system has 32GB
> RAM and with 3 workers it becomes visible. I wasn't monitoring memory
> consumption before.

There are some things that are shared between processes, such as listen 
socket for incoming RadSec connections, but otherwise the processes are 
independent. I'd say this is not a FarmSize issues, but I would not rule 
it out yet.

You mentioned you did not monitor memory consumption before: Did you 
restart processes daily already then?

> The automatic restart is scheduled at 6:59. That is the lowest value in
> the graph (3.18GB), later every 2nd hour in 25minute memory consumption
> jumps by ~1.5GB, detail:
>      http://tomasek.cz/stuff/memusage_radiusd-day.png
> 
> I do not think it is related to user activity as the graph seems to jump
> by constant step every 2h. CPU load graphs:
>      http://tomasek.cz/stuff/cpuload_radiusd.png
> top user activity is at 12.

How do you refresh CRL files? Would this be something that happens every 
2 hours?

> I'm running Radiator as Czech eduroam proxy. I've about 450 peer RADIUS
> servers, mostly (400+) using RadSec. I'm using CRL checking, which is
> the main reason for its initial memory footprint but can hardly explain
> memory leaks.

It might be useful to see if, for example, CRL files are refreshed on 
the file system periodically and this corresponds to process size growth.

> I'm logging with trace 4 and rotating logs every one hour:

> That is only periodical activity configured - but it happens every hour.

This should not matter.

> I'm running version 4.23.43 on Debian Stretch.

> Total memory usage of the system:
> 	http://tomasek.cz/stuff/total-memory.png
> does not show growth so clearly as only isolated Radiator. There is some
> periodicity at 40minute of every hour?

Any periodic events that happen are relative to each radiusd start time. 
This may be configurable at some time in the future, but now things that 
happen periodically within a radiusd are relative.

For example, if Client list is refreshed from SQL every 50 minutes, the 
first refresh happens when radiusd has been running for 50 minutes.

> I will start monitoring also Resident Set Size, I guess it will not show
> steps like Virtual, but fact that radiator is leaking memory remains.

How does it look? Anything periodic looking happening there?

Thanks,
Heikki

-- 
Heikki Vatiainen <hvn at open.com.au>

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, TACACS+, PAM, Active Directory,
EAP, TLS, TTLS, PEAP, WiMAX, RSA, Vasco, Yubikey, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, etc.