Memory leaks on main process
Created by: M5oul
On my Internet Cube dedicated to Duniter node, there memory leaks as duniter process takes 750Mb. This process is running since five days at least. This process takes almost 130Mb when turning-it on.
That could explain why my not-dedicated ARM board was unrechable since there were no more memory available.
Other: I don't know why there is so much processes.
Activity
Created by: sachaz
On v1.2.2 Duniter is crashing every day with a Crash oom-killer I have tried to set the following kernel parameters with no effect:
vm.overcommit_memory = 2 vm.overcommit_ratio = 95 here are some info of my system NAME: MAximeGhesquiere Adress: duniter.aquilenet.fr OS: Debian / jessie Ram: MemTotal: 1019952 kB SwapTotal: 524284 kB Dmesg: [95270.413529] duniter_default invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [95270.413551] duniter_default cpuset=/ mems_allowed=0 [95270.413560] CPU: 2 PID: 6708 Comm: duniter_default Not tainted 3.16.0-4-amd64 #1 Debian 3.16.39-1+deb8u2 [95270.413567] 0000000000000000 ffffffff81514c81 ffff88000ea13330 0000000000000000 [95270.413574] ffffffff81512859 0000000000000000 ffffffff810d6e5f 0000000000000000 [95270.413580] ffffffff815195fe 0000000000000200 ffffffff81068a23 ffffffff810c43f4 [95270.413587] Call Trace: [95270.413598] [<ffffffff81514c81>] ? dump_stack+0x5d/0x78 [95270.413605] [<ffffffff81512859>] ? dump_header+0x76/0x1e8 [95270.413618] [<ffffffff810d6e5f>] ? smp_call_function_single+0x5f/0xa0 [95270.413624] [<ffffffff815195fe>] ? mutex_lock+0xe/0x2a [95270.413631] [<ffffffff81068a23>] ? put_online_cpus+0x23/0x80 [95270.413638] [<ffffffff810c43f4>] ? rcu_oom_notify+0xc4/0xe0 [95270.413646] [<ffffffff8115407c>] ? do_try_to_free_pages+0x4ac/0x520 [95270.413652] [<ffffffff81142b3d>] ? oom_kill_process+0x21d/0x370 [95270.413658] [<ffffffff811426fd>] ? find_lock_task_mm+0x3d/0x90 [95270.413663] [<ffffffff811432a3>] ? out_of_memory+0x473/0x4b0 [95270.413674] [<ffffffff8114916f>] ? __alloc_pages_nodemask+0x9ef/0xb50 [95270.413682] [<ffffffff8118866d>] ? alloc_pages_current+0x9d/0x150 [95270.413688] [<ffffffff811418a0>] ? filemap_fault+0x1a0/0x420 [95270.413694] [<ffffffff81167aca>] ? __do_fault+0x3a/0xa0 [95270.413700] [<ffffffff8116a68e>] ? do_read_fault.isra.54+0x4e/0x300 [95270.413707] [<ffffffff81005479>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [95270.413713] [<ffffffff8116bebc>] ? handle_mm_fault+0x63c/0x11c0 [95270.413720] [<ffffffff810582c7>] ? __do_page_fault+0x177/0x4f0 [95270.413726] [<ffffffff81172ac9>] ? do_mmap_pgoff+0x2e9/0x3b0 [95270.413736] [<ffffffff8118f6b8>] ? kmem_cache_free+0xd8/0x210 [95270.413741] [<ffffffff81005012>] ? xen_mc_flush+0x192/0x1c0 [95270.413747] [<ffffffff81004632>] ? xen_clts+0x72/0x180 [95270.413753] [<ffffffff8151ce68>] ? page_fault+0x28/0x30 [95270.414434] Mem-Info: [95270.414445] Node 0 DMA per-cpu: [95270.414453] CPU 0: hi: 0, btch: 1 usd: 0 [95270.414458] CPU 1: hi: 0, btch: 1 usd: 0 [95270.414463] CPU 2: hi: 0, btch: 1 usd: 0 [95270.414468] CPU 3: hi: 0, btch: 1 usd: 0 [95270.414478] Node 0 DMA32 per-cpu: [95270.414482] CPU 0: hi: 186, btch: 31 usd: 0 [95270.414488] CPU 1: hi: 186, btch: 31 usd: 0 [95270.414493] CPU 2: hi: 186, btch: 31 usd: 30 [95270.414499] CPU 3: hi: 186, btch: 31 usd: 0 [95270.414506] active_anon:120900 inactive_anon:120977 isolated_anon:32 active_file:40 inactive_file:876 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 free:1979 slab_reclaimable:2055 slab_unreclaimable:2078 mapped:637 shmem:1 pagetables:1585 bounce:0 free_cma:0 [95270.414533] Node 0 DMA free:3968kB min:60kB low:72kB high:88kB active_anon:5488kB inactive_anon:5732kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:156kB kernel_stack:32kB pagetables:104kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [95270.414559] lowmem_reserve[]: 0 978 978 978 [95270.414567] Node 0 DMA32 free:3948kB min:3968kB low:4960kB high:5952kB active_anon:478112kB inactive_anon:478176kB active_file:164kB inactive_file:3508kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:1032192kB managed:1004040kB mlocked:0kB dirty:0kB writeback:0kB mapped:2504kB shmem:4kB slab_reclaimable:8164kB slab_unreclaimable:8156kB kernel_stack:1376kB pagetables:6236kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:14112 all_unreclaimable? yes [95270.414593] lowmem_reserve[]: 0 0 0 0 [95270.414602] Node 0 DMA: 1*4kB (U) 16*8kB (UM) 12*16kB (UM) 10*32kB (UEM) 8*64kB (UEM) 2*128kB (UM) 2*256kB (UM) 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 3972kB [95270.414629] Node 0 DMA32: 74*4kB (UE) 0*8kB 1*16kB (R) 0*32kB 2*64kB (R) 1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 4152kB [95270.414655] 5398 total pagecache pages [95270.414661] 4444 pages in swap cache [95270.414666] Swap cache stats: add 476945, delete 472501, find 147865/228710 [95270.414670] Free swap = 0kB [95270.414675] Total swap = 524284kB [95270.414679] 262047 pages RAM [95270.414683] 0 pages HighMem/MovableOnly [95270.414686] 7038 pages reserved [95270.414690] 0 pages hwpoisoned [95270.414694] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [95270.414715] [ 132] 0 132 7218 5 18 64 0 systemd-journal [95270.414724] [ 157] 0 157 10204 1 21 125 -1000 systemd-udevd [95270.414732] [ 329] 0 329 13796 0 30 180 -1000 sshd [95270.414741] [ 330] 0 330 6876 6 18 66 0 cron [95270.414748] [ 333] 0 333 4964 6 13 56 0 systemd-logind [95270.414761] [ 336] 105 336 10531 0 27 98 -900 dbus-daemon [95270.414770] [ 342] 0 342 1243 1 8 24 0 syslogd [95270.414778] [ 349] 0 349 3604 2 11 37 0 agetty [95270.414785] [ 350] 0 350 3559 2 12 39 0 agetty [95270.414795] [ 4695] 0 4695 7680 411 19 947 0 tmux [95270.414802] [ 4696] 0 4696 5483 2 15 143 0 bash [95270.414810] [ 4701] 0 4701 5492 2 14 153 0 bash [95270.414823] [ 5005] 0 5005 1461 8 8 19 0 tail [95270.414832] [ 5538] 106 5538 125179 135 29 235 0 stunnel [95270.414840] [ 6672] 0 6672 11590 2 26 95 0 su [95270.414848] [ 6673] 1000 6673 5474 3 15 131 0 bash [95270.414856] [ 6708] 1000 6708 643913 237136 1286 127172 0 duniter_default [95270.414865] Out of memory: Kill process 6708 (duniter_default) score 946 or sacrifice child [95270.414877] Killed process 6708 (duniter_default) total-vm:2575652kB, anon-rss:947044kB, file-rss:1500kB [101319.708527] nr_pdflush_threads exported in /proc is scheduled for removal [101319.708854] sysctl: The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case. If you have one, please send an email to linux-mm@kvack.org.Created by: sachaz
for the moment to correct the problem I have this script:
cat /etc/cron.hourly/duniter_check #!/bin/bash Test=`su -c "duniter status" duniter` if [ "$Test" = "Duniter is not running." ]; then date "+%Y-%m-%d %H:%M" >> /var/log/duniter_restart su -c "duniter webstart" duniter fi2 things:
High memory usage
Duniter process can take up to 600Mb or more memory without any memory leak: the garbage collector is not always acting. So at a given time you can have Duniter taking 400Mb of memory and a minute later taking only 100Mb. That's completely possible, I can observe it on my personal server.
Memory leak on member nodes
However I detected an obvious memory leak which is happening only on member nodes. That is, nodes computing blocks.
The leak is here: the
exchangesmap is fed over and over, and never cleaned. So it is always growing.I will fix this leak for next release.
Created by: zorun
I am having the same issue with 1.2.2 on x86_64 with the release Debian package: the amount of RAM used by duniter is growing steadily over time. It looks like a memory leak.
I was using v1.1.0 between 17 March and 7 May (without any restart) and it was not using a lot of RAM. Now, 24 hours after the upgrade to v1.2.2, it's already using 900 MB of RAM.
This node is only a mirror (it does not compute any block).
Created by: msand
@c-geek Would it be sufficient / help to change https://github.com/duniter/duniter-prover/blob/master/lib/engine.js#L43-L48 from
} else if (exchanges[msg.uuid].isFulfilled()) { logger && logger.error('PoW engine has sent a message about an already fulfilled uuid:'); logger && logger.debug(msg); } else { exchanges[msg.uuid].extras.resolve(msg.answer); }to
} else { exchanges[msg.uuid].extras.resolve(msg.answer); delete exchanges[msg.uuid]; }(Looking at the source code for the first time, so no proper insight into how things connect together yet...)
@msand Yes that's a good solution. I will also lock the problem with the following:
Instead of:
const exchanges = {};Set:
const exchanges = {}; // Permanently check the `exchanges` variable setInterval(() => { const keys = _.keys(exchanges) for (const k of keys) { if (exchanges[k].isFulfilled()) { delete exchanges[k] } } }, 10000)I think your solution should be enough, but I prefer to add safeguards just in case :-)
Finally released today!
https://forum.duniter.org/t/duniter-1-2-3-fuite-memoire/2510
