Vault balance top placeholders cause TPS peaks when being updated

[niall7459]

Vault balance top placeholders cause TPS peaks when being updated

[Niall7459]

I will look into it

[Bricec6]

Hello, I invite you to have a good look at #34 , I made several tests, remove all the holograms, put only the hologram of the balanctop and it is very real, the hologram including the placeholder of vault balance top, generates peaks of lags, when we remove the placeholder of the balancetop, and that we put back the others, there are no more peaks of lags.

If you don't find the mistake, he was defrauding the Essentials, or vault, from where the trouble could come. For now it will be impossible to use this type of hologram.

[Paul19988]

Yeah, just Bal Top hologram causing the issues and we last updated Vault Expansion on May 3rd

[MamaBeau]

Yeah, just Bal Top hologram causing the issues and we last updated Vault Expansion on May 3rd

Correct ^ once the holo was removed there were no more issues from then on out with lag of any kind.

[Niall7459]

Only when they are near? Can you do a timings report for me?
The issue will be investigated when a little more information is available.

Also see #34

[MamaBeau]

https://timings.aikar.co/?id=a0d26a1f1ec244778373b142b21ffb38

[MamaBeau]

Mind you, this is in my spawn which is in the FLATWORLD - that timings has been running for a while as the entire server almost crashed when I moved the balance top display to another location.

The second I remove the display from the server people can warp and go to spawn without lag or any delay. The second its back it takes 20 seconds for you to load in.

[Bricec6]

i'ts same problem for me, normally if you deviate from the balancetop hologram, your spike concerns will be solved. look #34

[MamaBeau]

I don't want to remove a top leader board, obviously thats not an ideal nor helpful fix

[Bricec6]

no I'm not talking about deleting it, I'm talking about deviating from the placeholder balancetop, if you load it your server will have peaks of tps, I'm just asking you to test it but normally that's it

[Bricec6]

sorry i'im french

[MamaBeau]

Theres no other placeholder to track top balance? I'm not sure what you are saying.

[Bricec6]

look at the link I put above my post, I show a little bit where the lag comes from, I look at your timings, I see that you with a lot of tick entities that causes you a big lag speak

[Bricec6]

[MamaBeau]

Thats not whats causing the lag at my spawn however, I've been testing a ton with whats causing the issues with my players and this plugin will literally take my server down to 13 TPS when I move or change that scoreboard.

I started by removed entities and that didn't fix the issues, then I stated by removing plugin, starting with Holographic Displays and Holographic Exten - this single handedly fixed the issues at my spawn. People could come to spawn and were saying how happy they were that it was fixed.

So not to lose this plugin that Ive used for so many years, I removed the holos one by one until I found the one causing the issues, which was the balance one.

I'm not here for any other reason other than the fact that the scoreboard for balance top is whats rendering my spawn useless.

[Bricec6]

so I have the same problem, ;)

[Niall7459]

What refresh holder did you give them? {medium}, {fast} e.g?

[Bricec6]

{slowest} for me

[MamaBeau]

It doesn't matter what refresh I use - we tried them all and they all lag still

[MamaBeau]

Any update to this?

[Bricec6]

thanks

[Bricec6]

hello do you have a news ? :D

[MamaBeau]

Yeah, I'm getting dangerously close to launching my new season and would have loved some progress on this issue

[Bricec6]

same for me ..

[MamaBeau]

Just looking for some kind of insight before I switch to another plugin.

[Niall7459]

I havn't had time to properly look at this, I still don't see a reason why this happens of top of my head.

The placeholders you are using.. are they accessing databases or anything on the fly?
Because this is vault vault is only the api. What's your underlying economy implementation, Essentials e.g.

[MamaBeau]

Not accessing a database
Essentials eco

Using the vault placeholder obviously.

What it seems like is maybe not this plugin but maybe the way essentials is refreshing the balances, I know another person that is using the board on their server without an issue and using essentials. They have a small playerbase though, where I have 3k unique joins.

[MamaBeau]

Anything?

[Niall7459]

The lag spikes don't seem consistent. Infact they are over 4 minutes and not equally spaced

GC PS MarkSweep: 35 avg(628.09ms every 1416.26s)
Wrong Garbage Collector FIX THIS
Switch Java flags to stop receiving lag spikes. https://aikar.co/2018/07/02/tuning-the-jvm-g1gc-garbage-collector-flags-for-minecraft/

[MamaBeau]

we've done that, and that timings report isnt very accurate for us because somewhere in there the board was removed. I couldn't continue to run it and it almost crash my server, had to remove it. Since removing it, we've not had a single. Not one. Lag spike.

[MamaBeau]

I guess the only real thing I can do is to switch plugins. And add LeaderHeads to track this and see if it happens with that one.

Its sad that we're still at the point of questioning if this is real or not with this plugin, after almost 3 weeks. :(

[Niall7459]

The main issue, as with any software, is narrowing down the cause of this. I know Brice already gave me a configuration of his holograms, could you provide yours please?

What happens if you replace the vault placeholders (all 20 of them) with something such as %player_time%. Do you still observe the TPS peaks?

[Niall7459]

I checked out repository for the vault expansion, all seems fine, it's updating placeholders async so shouldn't cause any disruption, unless your running on a single core system.

That leaves us with holographic extension unless your results from swapping placeholders say different. As for holographic extension there is not much room for performance improvement's because of the way it works. But we will see

[Paul19988]

The issue here could possibly be with Essentials eco, is it possible that it does not update BalTop Async which then causes a large delay inbetween the systems communicating?

[MamaBeau]

We discovered that there's a similar issue with that use Envoy's plugin and a timer. That Holo extension updates too fast for the timer, but we aren't sure if thats going to cause us any lag.

I was running 2 other leaderboards at my spawn on that server, island top (BentoBox) and VoteTop (VoteParty) they didn't lag the server at all.

We cant currently test the results of adding a different vault placeholder because our server is no longer running as we reset today at 1pm - but what @Paul19988 said makes sense (Paul is my developer)

[Niall7459]

The issue here could possibly be with Essentials eco, is it possible that it does not update BalTop Async which then causes a large delay inbetween the systems communicating?

This is what I initially thought, the trail is actually quite long

HolographicExtension -> PlaceholderAPI -> Vault Expansion -> Vault -> Essentials Eco.
There is actually a task in the Vault Expansion updating and caching the BalTop placeholders every 30 seconds.

This was added in March, relatively new
PlaceholderAPI/Vault-Expansion@a4514ea

That leads me to another thing- have you got the latest version of the Vault Expansion?

[Niall7459]

We discovered that there's a similar issue with that use Envoy's plugin and a timer. That Holo extension updates too fast for the timer, but we aren't sure if thats going to cause us any lag.

I was running 2 other leaderboards at my spawn on that server, island top (BentoBox) and VoteTop (VoteParty) they didn't lag the server at all.

We cant currently test the results of adding a different vault placeholder because our server is no longer running as we reset today at 1pm - but what @Paul19988 said makes sense (Paul is my developer)

Sorry, are the VoteTop and IslandTop leaderboards also using HolographicExtension? And it's just the Bal Top hologram causing issues?

[Niall7459]

That doesn't leave me with much, are you able to run another timings report for slightly longer

[MamaBeau]

I have no server to do that on, as stated above.

We had 50 players on with advertisement one day, and it took my TPS down to 8.
Removed the holos at spawn - TPS shot back up to 20.

If you would have been about a week sooner I could have gotten you another timings report that was more accurate but since we are resetting today, that server is no longer in function.

[Niall7459]

Of course it's difficult for me to perform accurate tests in a different server environment.

I will install a similar setup to what your using and run a profiler on the server to see if I can more accurately narrow down the issue.

Using a profiler may be able to give me a better insight on the issue. How many offline/unique players did you have on your server?

And how many would you usually have on at one point when issues occured?

[Bricec6]

You can make the test with no player ;)
You just have to be close to the hologram for it to be active in the chunk. It will generate spikes.
That's what I did, you can test, if you move away from the hologram and the chunk isn't loaded (move away, reboot the server) and do a test you won't see any spikes.

[Niall7459]

Ok thank you, I'll run this test now

[Niall7459]

Just for the sake of it what server version are you running?

[MamaBeau]

Pauls going to throw our server up, the old one with all the data if you want to come see it.

1.15.2

[Bricec6]

1.15.2

[MamaBeau]

If you want to give me your in game names I can whitelist you both and we can check it out together

[Niall7459]

I have ran some tests

I think I am able to safely rule out the Vault expansion, essentials and placeholder api, although this takes into account 1 single offline player, considering your server had 3k offline players.

I'm left with HolographicExtension and Vault

Although they are still only using 2.2% of the overall load, this may be a lot more visible when it's confined to a short space of time (tps peaks)

It may just be because updating 20 placeholders via packet injection is too intensive

[MamaBeau]

We found the issues actually - I think paul is going to type it up

[MamaBeau]

We only had 10 not 20 btw

[Paul19988]

Heyo, so after pulling apart the holos & playing around with them (deleted them all & reinstalled the plugins + expansions) It appears to have solved the issues

[Niall7459]

Ok, any more detail on that, did you remove any player files?

[Paul19988]

Did not remove any players files, quite literally just reinstalled the extensions & plugins, could have just been the case of a failed upload previously

[Niall7459]

Interesting, I'm putting my bets on the vault expansion update failing or something. Glad it fixed for you @Bricec6 any update on your issue?

[Bricec6]

i have update expansion, I had updates with /papi ecloud status, it's ok now, everything works

[Bricec6]

Hello @Niall7459 i have new
i have this lag check ;)
i think its error of this issue

https://pastebin.com/aBY9RVGa

config: https://pastebin.com/tZURrNLv

[Bricec6]

up ?

[Niall7459]

I believe this is an issue with "Dkcoins" placeholders

java.lang.IndexOutOfBoundsException: Index: 2, Size: 1
    at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_252]
    at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_252]
    at ch.dkrieger.coinsystem.spigot.hook.PlaceHolderAPIHook.onPlaceholderRequest(PlaceHolderAPIHook.java:62) ~[?:?]

You can fix this by removing the placeholders, but its worth contacting the author to see what the issue is