Heavy Memory load-1.9G memory usage with 30 clients.

[notgiven688]

Heavy Memory load-1.9G memory usage with 30 clients.

[notgiven688]

This was on the test-servers - where I tried to take a fee which perfectly pays the server costs - so no ;)

you use ubuntu 16.04? which version of mono?

@letlet "run webminerpool behind nginx" -> You use nginx as a proxy?

[notgiven688]

I will implement a dynamic Jobq. soon. At the moment it is limited to 100k entries. To test if it has any effect modify the line

public const int JobCacheSize = (int) 1e5;

in Program.cs. Try

public const int JobCacheSize = (int) 100;

[notgiven688]

@nadal28 You can do that. But it should be much more efficient to directly connect the server with your clients.

[notgiven688]

@letlet That is good news. Just set a lower value. I will be a dynamic value in the new releases. Thank you for testing.

[develth]

With the latest update, the RAM usage on both server is pretty awesome.
No increase - all as it should.
Thanks

[letlet]

Memory usage still growing even no new client. Maybe this problem is with jobqueue?
Active: active (running) since Tue 2018-05-15 14:07:42 CST; 1 day 18h ago
Main PID: 32101 (mono)
Tasks: 17
Memory: 3.0G
CPU: 3min 56.442s
May 17 08:44:19 ubuntu mono[32101]: [5/17/2018 8:44:19 AM] heartbeat; connections client/pool: 9/2; jobqueue: 62.1k; speed: 0.7kH/s

[Mila432]

i think so , it eats 29gb ram on my server
but i also have more connections

[letlet]

29Gb is a huge number, I used to have another proxy(before CN v7) made by golang which can host over 20K clients with less than 1Gb memory usage.
Must be memory leak some where with this proxy.

[notgiven688]

Hui.. never had this problem. 40k connections - 2 gbyte.

[nadal28]

40k? are you rich ? xD @notgiven688

[barrettathome]

@nadal28 I may be experiencing a similar issue. Running a small Ubuntu server with 8 to 12ish connections. After being online for a while a "killed" message would appear and I would have to restart the service. It does seem that the "jobqueue" keeps growing in size.

[nadal28]

is not "good" to use a web server proxy?? Im using an apache proxy just to use the SSL cert of my domain

[letlet]

@notgiven688 Yes, I use the nginx as proxy server because I need the 443 port for some other websites in the same server. And yes it's Ubuntu 16.04.
Here is the mono on my server:

Mono JIT compiler version 5.12.0.226 (tarball Thu May 3 09:48:32 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: supported, not enabled.
GC: sgen (concurrent by default)

Today the jobqueue reached 100.0k and it cost 5.3G , I will report the momory usage after a while to see if jobqueue's size connected with huge memory

[letlet]

Still 5.3Gb memory usage with 100.0k jobqueue after 2 days. So the problem should be the jobqueue.
I have no more than 30 clients concurrent running, why so large jobqueue?

[letlet]

@notgiven688 Thank u for your excellent work, I will test the new version when released and feedback here.

[brainscar]

@letlet what is your memory usage now?

[letlet]

@brainscar No improvement now.

[letlet]

I have upgraded mono to:
Mono JIT compiler version 5.12.0.301
and webminerpool server cost about 650M memory with about 1000 clients after 16 hours's running.
I upgraded some other packages too(with apt upgrade).
So, maybe it's mono's problem or even the local enviroment's problem?
I'll continue investigate and update here.

[notgiven688]

@letlet Thank you for your tests/reports. I need some brave men, since I can not reproduce this stupid memory problem... @everyone: feel free to write about your memory consumption here.

[letlet]

webminerpool has runned stably for last 8 days and cost about 3.5G memory with 1000 clients since day 2.

Then it cost 5.0G memory with 1000 clients after a restart 12 hours ago.

I will try another server soon later to verify it this connected with the server's env.

[notgiven688]

@letlet This is too much. 1000 clients should use less than 1gb. Maybe you can try to run the docker file..

[marcelobcortes]

having such massive amount of workers in a single server is a reliable choice?
well if i get constantly 40k workers i'd for sure spin up a pool
but i wonder if some minor or major issues could happen in this single server architecture that could make me cry

can you tell me your server specs and estimated bandwidth in that 40k situation?

[develth]

I have about 4 users and it always grows pretty fast to 3G. Have to restart it almost every two days.
Over the current night just one - but still growing to max memory

[nox956]

In my case, disable hash check (libhash.so) solve high mem usage problem.

[develth]

thanks for the hint, @nox956
I removed the libhash.so from one of the two servers, so i can compare if it will change the memory consumption

[notgiven688]

@develth Result would be very interesting because it indicates a memory leak in libhash. It was rewritten for cn_v2 anyway, but I am curious.

[develth]

Currently both have a memory consumption of 70%

[develth]

Just for "Time-Log-Reporting":
Today i got a mail from my monitoring - currently server.exe uses 80% of 4GB RAM with currently 2 Clients (before between 12 and 0) and does not shrink after clients disconnects.
I will leave it until it goes up to 90% and will report again.

[develth]

Now at 90%, still only 2 Clients

[notgiven688]

@develth Latest update = latest system update? Or latest update of webminerpool?

Thanks for reporting - as always :)

[develth]

Latest webminerpool update