Specify some minimum system requirements in Readme

[lmachens]

Specify some minimum system requirements in Readme

[lmachens]

This is a difficult question. My server has 1.5 GB memory and 2 cores. It runs a single mongo replica set and the apm server. After about 2 weeks the UI and aggregation is getting very slow. It helps to clean up the database (I don't need data older than 2 weeks, so it is fine for me).
I have 6 servers and 1 application with 200 to 500 users online at the same time. So there are a lot of data generated.
The big performance problem is the incremental aggregation which takes longer and longer after a while. Problems starts when the aggregations takes longer than the interval. I am sure that it is possible to handle it.

I would recommend to use at least two cores if you use a single server for the apm and database. Memory is not so important (500mb should be fine).

[markreid]

I've got it running on a single core Lightsail instance and with more than about 2-3 weeks worth of data it becomes unusable because the aggregation methods are called faster than they can complete.

I saw your todo note to configure a timeout for pruning old data @lmachens, have you had a chance yet to look at exactly what the aggregation is doing and whether there's any optimisations to be made there? It seems odd that the entire historical data set needs to be crunched every X minutes...

[lmachens]

@markreid Yes, I know that problem. I think the best way is to optimize the query and indexes. The aggregation is incremental.

async function runAll() {
  await incrementalAggregation(PROFILES['1min'], PROVIDERS['errors']);
  await incrementalAggregation(PROFILES['1min'], PROVIDERS['methods']);
  await incrementalAggregation(PROFILES['1min'], PROVIDERS['pubsub']);
  await incrementalAggregation(PROFILES['1min'], PROVIDERS['system']);
  await incrementalAggregation(PROFILES['30min'], PROVIDERS['errors']);
  await incrementalAggregation(PROFILES['30min'], PROVIDERS['methods']);
  await incrementalAggregation(PROFILES['30min'], PROVIDERS['pubsub']);
  await incrementalAggregation(PROFILES['30min'], PROVIDERS['system']);
  await incrementalAggregation(PROFILES['3hour'], PROVIDERS['errors']);
  await incrementalAggregation(PROFILES['3hour'], PROVIDERS['methods']);
  await incrementalAggregation(PROFILES['3hour'], PROVIDERS['pubsub']);
  await incrementalAggregation(PROFILES['3hour'], PROVIDERS['system']);
}

Meteor.startup(() => {
  runAll();
  setInterval(runAll, 60000);
});

I think 1min depends on raw, 30min depends on 1min and 3hours depends on 30min.
The 1min should be fast, because the raw data is removed after the aggregation.

  if (SourceCollection === PROVIDER.rawCollection) {
    // Clean rawCollection
    PROVIDER.rawCollection.remove(query);
  }

I would take a deeper look to MapReduce(SourceCollection, DestCollection, PROVIDER.map, PROVIDER.reduce, options); and check if the mongo indexes makes sense. Maybe there is more data we can remove directly after the aggregation.

[lmachens]

I did some improvements to the indexes and cleanup.
In addition, you can set the maximum lifetime of metrics in the settings.json.
Example (default 7 days ^= 604800000) :
"metricsLifetime": 604800000

Please give me feedback if this changes makes the apm server more stable.

[cwohlman]

The changes look good, I haven't been the server running long enough to detect issues.

[cwohlman]

For what it's worth I've been running the server on a t2-micro for non-production loads (my app has fewer than 50 concurrent users) & it's working fine so far.