Daemon crashed with FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[noisy]

The problem:

Daemon crashed

I did

wise deamon

I expected to see:

Daemon running

Instead, I saw:

Links to gist with output / logs (or at least a picture):

Processed block 29063979
Processed block 29063980
Processed block 29063981
Processed block 29063982
Processed block 29063983
Processed block 29063984
Processed block 29063985
Processed block 29063986
Processed block 29063987
Processed block 29063988

<--- Last few GCs --->

[30379:0x4115510] 1888719574 ms: Mark-sweep 1268.2 (1437.3) -> 1268.2 (1437.3) MB, 1122.9 / 0.0 ms  allocation failure GC in old space requested
[30379:0x4115510] 1888720978 ms: Mark-sweep 1268.2 (1437.3) -> 1268.2 (1437.3) MB, 1262.8 / 0.0 ms  last resort GC in old space requested
[30379:0x4115510] 1888722245 ms: Mark-sweep 1268.2 (1437.3) -> 1268.2 (1437.3) MB, 1267.0 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2e9e32aa55e9 <JSObject>
    0: builtin exit frame: parse(this=0x2e9e32a88e19 <Object map = 0x141c76f02ba1>,0x107a040d1d01 <Very long string[17499]>)

    1: /* anonymous */ [/usr/local/lib/node_modules/steem-wise-cli/node_modules/node-fetch/lib/body.js:46] [bytecode=0x147d13918c11 offset=27](this=0x3cccdbc066f1 <JSGlobal Object>,buffer=0x107a040a2d39 <Uint8Array map = 0x141c76f59e91>)
    2: /* anonymous */(aka /* anon...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x8d04bc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
 7: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [node]
 8: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::PretenureFlag) [node]
 9: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [node]
10: v8::internal::JsonParser<false>::ParseJsonInternalizedString() [node]
11: v8::internal::JsonParser<false>::ParseJsonObject() [node]
12: v8::internal::JsonParser<false>::ParseJsonValue() [node]
13: v8::internal::JsonParser<false>::ParseJsonObject() [node]
14: v8::internal::JsonParser<false>::ParseJsonValue() [node]
15: v8::internal::JsonParser<false>::ParseJson() [node]
16: v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::internal::Isolate*) [node]
17: 0x2e66bf08697d
Aborted

Version of nodejs: 3.1.0-rc.1
Node version: v9.11.1

[noisy]

@Jblew it happened again:

Processed block 29334804

<--- Last few GCs --->

[24268:0x343d510] 1889725276 ms: Mark-sweep 1269.3 (1437.8) -> 1269.3 (1437.8) MB, 1056.7 / 0.0 ms  allocation failure GC in old space requested
[24268:0x343d510] 1889726638 ms: Mark-sweep 1269.3 (1437.8) -> 1269.3 (1437.8) MB, 1200.3 / 0.0 ms  last resort GC in old space requested
[24268:0x343d510] 1889727852 ms: Mark-sweep 1269.3 (1437.8) -> 1269.3 (1437.8) MB, 1213.8 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x34636ea55e9 <JSObject>
    2: oncomplete(aka afterConnect) [net.js:1137] [bytecode=0x26d047adbe09 offset=0](this=0x264f1dfede21 <TCPConnectWrap map = 0x7bc359fc541>,status=0,handle=0x264f1dff3879 <TCP map = 0x7bc359fd671>,req=0x264f1dfede21 <TCPConnectWrap map = 0x7bc359fc541>,readable=0xe6194b82371 <true>,writable=0xe6194b82371 <true>)

==== Details ================================================

[2]: oncomplete(aka...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x8d04bc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [node]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
 7: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
 8: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
 9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x3c02b23042fd
Aborted

[Jblew]

@noisy Apparently there is a momory leak. v8 (node interpreter) has a strict memory limit of 1.7GB, so it means that wise-cli peaked to that! / GC in v8 has some limitations. Probably somewhere in code is some often invoked closure, or unnecessary multiple references to an object. There are memory leak detection tools for node, we'll have to run them.

[noisy]

I've made an upgrade of node to version 11.6.0. We will see if this will help...

[Jblew]

@noisy 11.x is not supported nor tested. I would recommend to use the latest LTS (which is 10.x). Primarily in all tests, docker and on our server we use 10.12. We should probably upgrade to 10.15. There are few mwmory leak-related entries in node v8's change log between these versions

[bgornicki]

@noisy why don't you use wise in a docker container?

[noisy]

@bgornicki because I wanted to debug this problem, not find a workaround for it.

[bgornicki]

And a Docker container is a solution for creating strictly controlled environment, perfect for finding and reproducing bugs like that ;)