Datastore/cache reduces performance.

[yrsh]

When we load more than 500 rps, the performance falls heavily. Because the datastore-logic (apiloader middleware) starts sending over the network snapshots data between microgateway, datastore and cache. What to do with it, can it be somehow optimized or disabled? Why it sends every time http-request (in getCurrentSnapshot) on every request to the api?

[tonyffrench]

Lets work on a PR to address this issue. Options as I see it 1) use HTTP caching 2) Use an async notification based caching approach, whereby µGW will only contact the datastore when needed 3) Move the datastore in-process (need to consider any clustering implications). @yrsh thoughts?

[yrsh]

Hi, Tony.
Thank you for your response. We've tried to implement point one, with results of increasing overall performance about 5 times for our system. We will be ready to provide you access to PR on Monday. 2 - yes, it can be an option, maybe we will use pub/sub approach. But here we have some questions regarding the solution architecture. Am I correct that datastore major purpose is to sync configuration between instances near realtime? If yes is it an option to use it terms of immutable container architecture? It could also provides the benefits of channels isolation .For example we would be able to apply different mashups configurations, depending on channel (i.e. mobile apps, IoT apps, partner systems). So, it means that each time we publish the configuration to particular channel we build new image and redeploys corresponding containers. Perhaps it could be useful to optionally disable the datastore completely? Regarding point 3 it's interesting, but we need to think a bit more.
Regards.

[tonyffrench]

I think your questioning the validity of the datastore approach assuming immutable containers (--read-only)? One option is to update the datastore to be an in memory cache and remove the dependency on the filesystem.

Lets work on delivering option one as a quick win and then pivot the conversation to whats optimal for immutable containers.

[MahdiMahava]

Sorry for the delay, we'll upload code tomorrow morning.

[yrsh]

Hi!
I forked microgateway and datastore repo. They depend on each other, so mgw uses forked datastore. Some changes are required in mgw repo as well. So how do we manage to deliver our changes on your codebase? Maybe we create new branches in each repo and link them after that in package.json, for example?

mgw:
https://github.com/yrsh/microgateway

datastore:
https://github.com/yrsh/microgateway-datastore

Results of load testing:

Left - without cache, right - with cache. Load generator - artillery.io. We used simple definition with javascript policy, which writes simple response text.

[tonyffrench]

@yrsh pls create a PR (https://help.github.com/articles/creating-a-pull-request-from-a-fork/) for each repo and I'll review the code and merge.

[yrsh]

@tonyffrench I've created several pull requests, but you are using very old node versions. Could you please comment why do you need this?

[tonyffrench]

We have customers using older LTS versions like v4 that are still active.

[yrsh]

@tonyffrench can we somehow go with ES6 or we need to remake our PRs to ES5?

[tonyffrench]

I've merged and published microgateway-datastore to npm at @1.0.2. Yes you need to make sure your changes are compatible with the latest LTS v4. cc @yrsh

[yrsh]

@tonyffrench I've created a new PR, with ES5:
#101

[yrsh]

We have already made performance improvements in flow-engine and now going to rewrite the datastore and then make PR with global changes. We need to know for which purpose is datastore used? As I see the key features are the synchronization of definitions and TLS profiles, but does store have any another important functionality?

[devanshah2]

Are there plans to have a fix for this.

Currently we are faced with an issue where if we have more then 3 to 4 spec files that need to be loaded into microgateway it hangs or runs out of memory when starting up. The out of memory looks to be coming from datastore.

Following is the error stack:

<--- Last few GCs --->

  776893 ms: Mark-sweep 1309.1 (1404.6) -> 1308.5 (1404.6) MB, 243.7 / 0.0 ms [allocation failure] [GC in old space requested].
  777217 ms: Mark-sweep 1308.5 (1404.6) -> 1308.5 (1404.6) MB, 292.1 / 0.0 ms [allocation failure] [GC in old space requested].
  777518 ms: Mark-sweep 1308.5 (1404.6) -> 1314.1 (1403.6) MB, 300.8 / 0.0 ms [last resort gc].
  777810 ms: Mark-sweep 1314.1 (1403.6) -> 1319.7 (1403.6) MB, 292.5 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x44728cf781 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/Users/devans@ca.ibm.com/Documents/dsoc/dsoc-apigateway/microgateway/src/node_modules/microgateway/node_modules/json-refs/index.js:~219] [pc=0x16be01a93ebf] (this=0x4472804381 <undefined>,refPtr=0xe86d5d719f9 <String[47]: #/paths/~1v1~1asset-statistics/get/parameters/2>)
    2: arguments adaptor frame: 3->1
    3: InnerArrayForEach(aka InnerArrayForEac...


<--- Last few GCs --->

  776893 ms: Mark-sweep 1309.1 (1404.6) -> 1308.5 (1404.6) MB, 243.7 / 0.0 ms [allocation failure] [GC in old space requested].
  777217 ms: Mark-sweep 1308.5 (1404.6) -> 1308.5 (1404.6) MB, 292.1 / 0.0 ms [allocation failure] [GC in old space requested].
  777518 ms: Mark-sweep 1308.5 (1404.6) -> 1314.1 (1403.6) MB, 300.8 / 0.0 ms [last resort gc].
  777810 ms: Mark-sweep 1314.1 (1403.6) -> 1319.7 (1403.6) MB, 292.5 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x44728cf781 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/Users/devans@ca.ibm.com/Documents/dsoc/dsoc-apigateway/microgateway/src/node_modules/microgateway/node_modules/json-refs/index.js:~219] [pc=0x16be01a93ebf] (this=0x4472804381 <undefined>,refPtr=0xe86d5d719f9 <String[47]: #/paths/~1v1~1asset-statistics/get/parameters/2>)
    2: arguments adaptor frame: 3->1
    3: InnerArrayForEach(aka InnerArrayForEac...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 4: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handle<v8::internal::String>, int, unsigned int) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 5: v8::internal::InternalizedStringKey::AsHandle(v8::internal::Isolate*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 6: v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::HashTableKey*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 7: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 8: v8::internal::LookupIterator::LookupIterator(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::LookupIterator::Configuration) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 9: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
10: v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
12: 0x16be012092a7
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 4: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handle<v8::internal::String>, int, unsigned int) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 5: v8::internal::InternalizedStringKey::AsHandle(v8::internal::Isolate*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 6: v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::HashTableKey*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 7: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 8: v8::internal::LookupIterator::LookupIterator(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::LookupIterator::Configuration) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
 9: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
10: v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/devans@ca.ibm.com/.nvm/versions/node/v6.14.3/bin/node]
12: 0x16be012092a7
Debug: datastore exited

[ozairs]

I have raised this issue within our Engineering team. We should have a response shortly.