Very high memory usage of `monGui`

[dpapagiannis]

From recent relval freezes, it has been shown that memory usage of monGui increases very quickly, taking up all available RAM (64GB) on vocms0737, leading to the machine being unresponsive and require a restart.

DQMGUI version currently running for relval: 10.2.1

Things to test:

• Simply update version; not much change is expected, though, since not many important changes have been made.
• Profile the application (e.g. with valgrind). This seems to be already accounted for here.

cc: @arooshap

[dpapagiannis]

The following valgrind command is now in use (in server-conf-online.py):

server.instrument  = 'valgrind --tool=memcheck --num-callers=500 --error-limit=no --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose'

The problem is however, that there's only some output when the application starts, then there's no other sign of output, even when the application is stopped.

Additionally, since there is no real slowdown in the application's performance even when running it with valgrind, I have the suspicion that it's not really executing it with the tool.

The full command that monControl is using to execute monGui is:

setsid sh -c exec valgrind --tool=memcheck --num-callers=500 --error-limit=no --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose python /data/srv/HG2409a/sw/venv/bin/monGui /data/srv/current/config/dqmgui/server-conf-relval.py >> /data/srv/logs/dqmgui/relval/weblog.log 2>&1

which seems to be correct.

---

Adding --trace-children=yes seems to help follow forked processes, which are spawned by monGui:

server.instrument  = 'valgrind --tool=memcheck --num-callers=500 --error-limit=no --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --trace-children=yes'

[dpapagiannis]

I discovered that the daily maintenance cronjob was disabled in the relval flavor of the GUI (/data/srv/state/dqmgui/.nodaily existed). This cronjob's tasks include:

• Rotating log files (leads to huge files of several GBs)
• Cleaning up open DQMGUI sessions

Re-enabling it seems to lead back to healthy use of memory again. It could be related to the uncleaned DQMGUI sessions.

Not closing the issue for now, until this is verified.

PS:

ls -alF /data/srv/state/dqmgui/.nodaily

total 0
drwxr-xr-x.  2 cmsweb zh   	10 Jan 18 02:20 ./

I guess this was somehow created manually(?), but I have no recollection of it.
This file gets created and deleted (trap "rmdir $STATEDIR/.nodaily" 0 1 2 15) by the daily script, to temporarily disable possible multiple calls of the same script. In principle, .nodaily should have already been deleted after the call to the daily script, but for some reason it was left there, inhibiting further calls to daily.

[dpapagiannis]

dmwm/deployment#1356 has worked this far as a workaround.

[dpapagiannis]

Sources of high memory usage identified:

• The LRU cache used by the server to keep the 1024 last requested images, which also keeps the databytes as received from the database (probably the raw histogram bytes). A PR to try to work around this is implemented in Reduce LRU image cache's memory footprint #46.
• For each visDQMRender process, around 250MB of memory usage is observed, once they have operated for some time. This number does not fluctuate a lot, except for some spikes in memory usage during rendering.
• On building a message to send to the renderer process (in serverext.cc), yet another copy of databytes is done. Perhaps a solution with a shared_ptr and a string_view on top of it could somehow work around this copy.

Other thoughts:

• Reducing the number of renderers could also help with memory usage, if the situation is desperate.
• Reducing the size of the LRU cache could also be a workaround in memory-limited machines.