JS heap out of memory when generating lots of empty buffers

[mnpenner]

• Version: 10.7.0
• Platform: Linux
• Subsystem: Ubuntu 16.04

Node is crashing with this error:

<--- Last few GCs --->

[17673:0x3342600]    41204 ms: Scavenge 1396.8 (1423.3) -> 1396.1 (1423.8) MB, 2.5 / 0.0 ms  (average mu = 0.153, current mu = 0.096) allocation failure 
[17673:0x3342600]    41211 ms: Scavenge 1396.9 (1423.8) -> 1396.3 (1424.3) MB, 2.2 / 0.0 ms  (average mu = 0.153, current mu = 0.096) allocation failure 
[17673:0x3342600]    41218 ms: Scavenge 1397.1 (1424.3) -> 1396.5 (1425.3) MB, 2.5 / 0.0 ms  (average mu = 0.153, current mu = 0.096) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x161cbe2041bd]
    1: StubFrame [pc: 0x161cbe258d05]
Security context: 0x395a7281e6c9 <JSObject>
    2: /* anonymous */ [0x319c2e50fc29] [/home/me/Projects/ouid/src/node.js:~1] [pc=0x161cbe2ef341](this=0x319c2e50fd59 <Object map = 0x3de7d3a823b9>,exports=0x319c2e50fd59 <Object map = 0x3de7d3a823b9>,require=0x319c2e50fd19 <JSFunction require (sfi = 0x3bbca4740099)>,module=0x319c2e50fc91 <Module map = 0x3de7d3acd...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8b5f80 node::Abort() [node]
 2: 0x8b5fcc  [node]
 3: 0xab730e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xab7528 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xea5152  [node]
 6: 0xeb10aa v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 7: 0xeb1a14 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xeb4345 v8::internal::Heap::AllocateRawWithRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
 9: 0xe7c824 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
10: 0x111e1de v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
11: 0x161cbe2041bd 

With this code:

function ouid() {
    let buf = Buffer.allocUnsafe(16);
    return buf;
}

for(;;) {
    process.stdout.write(`\r${ouid().toString('hex')}`);
}

I could be wrong, but since I'm not actually holding onto any of those buffers, I believe Node or V8 should be cleaning up that memory for me.

It takes about 40-65 seconds to exhaust the memory.

[mscdex]

This has nothing to do with buffer. You can even remove the function entirely (just writing a simple static string in the loop) and the same problem will occur. Explicitly calling gc() after writing the string to stdout will allow the process to stay alive. Also, you can set --max-old-space-size=128 or something similarly lower to have it crash more quickly.

Simplified example:

$ node --expose-gc --max-old-space-size=128 test.js

// test.js
for (;;) {
  process.stdout.write('\rhello');
  //gc();
}

[mnpenner]

Ahah. Sorry. I tried removing as much code from my program while still reproducing the error, I guess I should have kept going.

I'm uncertain if this is a "bug" or not though. Shouldn't Node or V8 be monitoring how much memory is being consumed and perform some garbage collection before it runs out?

[mscdex]

It's most likely the tight loop. If you switch the for to a setInterval(), the process won't crash.

[mnpenner]

@mscdex That works. Thanks for your help! I guess I'll close this then.