I’m building integrations to sync API data to my DB. It works all fine with “small” amount of records (1K-10K) but when I tried to do a large set it crashed the app. I see that it crashed when it came to 13.700 records.
My server connect api is basically first looping the api to get all records, and when that finishes I’m repeating all the loaded api records to insert it to the db. It reached the inserting to the DB part and crashed there so I’m assuming that it was just too “heavy” to repeat / do stuff / insert into the db.
Does anyone have any advise on how to solve this? I’m reading on Google that I should set a higher memory limit to node to process the task, but I’m planning to run such tasks often, will it affect the main app processes/functioning? How can I make this more efficient?
This is the server connect api:
The error:
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0xb02930 node::Abort() [/usr/local/bin/node]
2: 0xa18149 node::FatalError(char const*, char const*) [/usr/local/bin/node]
3: 0xcdd16e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xcdd4e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xe94b55 [/usr/local/bin/node]
6: 0xea481d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
7: 0xea751e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
8: 0x121701d v8::internal::Deserializer<v8::internal::Isolate>::Allocate(v8::internal::SnapshotSpace, int, v8::internal::AllocationAlignment) [/usr/local/bin/node]
9: 0x121b283 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [/usr/local/bin/node]
10: 0x1219c40 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject() [/usr/local/bin/node]
11: 0x12144f8 v8::internal::ContextDeserializer::Deserialize(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
12: 0x1214676 v8::internal::ContextDeserializer::DeserializeContext(v8::internal::Isolate*, v8::internal::SnapshotData const*, bool, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
13: 0x1239ce8 v8::internal::Snapshot::NewContextFromSnapshot(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, unsigned long, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
14: 0xf489e5 v8::internal::Genesis::Genesis(v8::internal::Isolate*, v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
15: 0xf49818 v8::internal::Bootstrapper::CreateEnvironment(v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, v8::ExtensionConfiguration*, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
16: 0xcea976 v8::NewContext(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
17: 0xceae44 v8::Context::New(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
18: 0xaf33f1 node::contextify::ContextifyContext::CreateV8Context(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [/usr/local/bin/node]
19: 0xaf41d7 node::contextify::ContextifyContext::ContextifyContext(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [/usr/local/bin/node]
20: 0xaf47d9 node::contextify::ContextifyContext::MakeContext(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
21: 0x1569fec [/usr/local/bin/node]
[nodemon] app crashed - waiting for file changes before starting...
Community Page
Last updated:
Last updated: