Comments (12)
@rustatian Hi!
Thanks for answers. After updating RR to 2023.1.5, the test project no longer has memory leaks
from roadrunner-temporal.
Hey @CheAlex 👋🏻
The workflow worker is internal and exists in a single copy. It's not safe to restart WF often because PHP would have to restart currently running workflows due to lack of information about them (restart - clear state). We have fixed an internal PHP memory leak, so this shouldn't be the case here. Please update to the latest RR version (2023.1.5 and corresponding PHP-SDK: 2.5.1)
from roadrunner-temporal.
broken load balancing - basically only workflow worker works
It's not broken. Workflow workers handle all workflow activity, all activity coordination. So, it's expected that the load on a single WF worker in terms of exec's
would be greater than on activity workers. But this is a very light load, since the WF worker, as I mentioned before, is only a coordinator. The real workload would be on an activity worker's shoulders.
from roadrunner-temporal.
Since this is not a bug, I'm closing this issue, but not locking it. So you can continue to discuss/ask questions.
from roadrunner-temporal.
@rustatian Hi!
I modified the test project: https://github.com/CheAlex/temporalio-samples-php/tree/test-memory-leaking-and-cpu-sticking - it uses the latest version of php-sdk and RR
After running I got the following results:
php app.php simple-activity
:
+---------+-----------+---------+---------+---------+--------------------+
| PID | STATUS | EXECS | MEMORY | CPU% | CREATED |
+---------+-----------+---------+---------+---------+--------------------+
| 11224 | ready | 251 | 48 MB | 0.75 | 31 seconds ago |
| 11225 | ready | 251 | 48 MB | 0.75 | 31 seconds ago |
| 11226 | ready | 249 | 48 MB | 0.78 | 31 seconds ago |
| 11227 | ready | 249 | 47 MB | 0.75 | 31 seconds ago |
| 11240 | ready | 3,001 | 51 MB | 9.64 | 31 seconds ago |
+---------+-----------+---------+---------+---------+--------------------+
After the end of all workflows, the workflow worker CPU consumption reaches ~10%, and then slowly drops to zero - there is a CPU sticking effect.
2. php app.php simple-activity-exception
:
+---------+-----------+---------+---------+---------+--------------------+
| PID | STATUS | EXECS | MEMORY | CPU% | CREATED |
+---------+-----------+---------+---------+---------+--------------------+
| 10676 | ready | 500 | 59 MB | 1.34 | 38 seconds ago |
| 10677 | ready | 499 | 59 MB | 1.31 | 38 seconds ago |
| 10678 | ready | 501 | 59 MB | 1.34 | 38 seconds ago |
| 10679 | ready | 500 | 59 MB | 1.34 | 38 seconds ago |
| 10694 | ready | 3,414 | 119 MB | 50.68 | 38 seconds ago |
+---------+-----------+---------+---------+---------+--------------------+
After the end of all workflows, the workflow worker CPU consumption reaches ~50%, and then slowly drops to zero - there is a CPU sticking effect. The memory remains at 119 MB and does not fall.
3. php app.php simple-activity-timeout
:
+---------+-----------+---------+---------+---------+--------------------+
| PID | STATUS | EXECS | MEMORY | CPU% | CREATED |
+---------+-----------+---------+---------+---------+--------------------+
| 13339 | ready | 4 | 48 MB | 0.09 | 55 seconds ago |
| 13340 | ready | 4 | 47 MB | 0.09 | 55 seconds ago |
| 13341 | ready | 4 | 48 MB | 0.12 | 55 seconds ago |
| 13342 | ready | 4 | 48 MB | 0.12 | 55 seconds ago |
| 13355 | ready | 3,238 | 126 MB | 37.40 | 55 seconds ago |
+---------+-----------+---------+---------+---------+--------------------+
After the end of all workflows, the workflow worker CPU consumption reaches ~40%, and then slowly drops to zero - there is a CPU sticking effect. The memory remains at 126 MB and does not fall.
Is this normal, expected behavior?
from roadrunner-temporal.
Hey @CheAlex 👋🏻
and then slowly drops to zero - there is a CPU sticking effect
This is how it should be 😃, no worries, but thanks for pointing this out.
You can easily get information about CPU core load. But if you try to calculate the same information about a process - you have to take into account other processes and how they affect CPU as well. And then calculate how much CPU time your process consumes. So you'll see this effect. CPU usage for the process increases, then slowly decreases (because it uses less CPU time).
This is not the same as CPU load. This is how much CPU time (%) that particular process is using.
If you open utilities like top
, btop
, htop
and check the total CPU core load (or load across cores), you'll see that the CPU load drops immediately after you stop loading your workers.
from roadrunner-temporal.
@rustatian and what about memory in the second and third cases?)
from roadrunner-temporal.
Regarding the memory:
RR uses /proc/<pid>/statm
to count RSS memory consumption per process, this method is inaccurate, but we don't have more accurate way. So, the memory consumption might be slightly over the limit, not by tens of megabytes, but still inaccurate. More info here:
(24) rss %ld
Resident Set Size: number of pages the process has
in real memory. This is just the pages which count
toward text, data, or stack space. This does not
include pages which have not been demand-loaded in,
or which are swapped out. This value is
inaccurate; see /proc/pid/statm below.
https://man7.org/linux/man-pages/man5/proc.5.html
from roadrunner-temporal.
@rustatian checked, everything works as you said, thanks)
from roadrunner-temporal.
@CheAlex 👍🏻 You are welcome 😃
from roadrunner-temporal.
not by tens of megabytes
but in my examples just tens of megabytes: 119 MB, 126 MB
from roadrunner-temporal.
These are 2 different runs, it's not a measurement problem, you're running two different workflows, so the memory consumption is different.
By "tens of megabytes" I meant OS-specific tools vs. RR measurement.
from roadrunner-temporal.
Related Issues (20)
- [INTEGRATION] `v1.x` rr-t integration TODOs with `php-sdk`
- [🐛 BUG]: Ghost workers (pollers) in roadrunner version 2.7.5 HOT 5
- [🐛 BUG]: Metric `temporal_activity_schedule_to_start_latency` useless with RR? HOT 14
- [🐛 BUG?]: MaxConcurrentActivityExecutionSize ignored. HOT 6
- [💡 FEATURE REQUEST]: Support LocalActivities
- [🐛 BUG]: Incorrect worker pool recreating HOT 38
- github.com/stretchr/testify-v1.7.2: 1 vulnerabilities (highest severity is: 7.5) - autoclosed HOT 1
- [💡 CHORE]: Move all protos to the buf RR API
- [💡 FEATURE REQUEST]: Overwrite `client-name` and `client-version` in Go client to represent PHP-SDK HOT 3
- [💡 FEATURE REQUEST]: Statsd support for metrics HOT 3
- github.com/uber-go/tally/v4-v4.1.2: 2 vulnerabilities (highest severity is: 7.5) - autoclosed HOT 1
- [🐛 BUG]: Worker after SIGTERM not wait for finish activity HOT 4
- [💡 FEATURE REQUEST]: Support FailureConverter API HOT 2
- github.com/uber-go/tally/v4-v4.1.3: 2 vulnerabilities (highest severity is: 7.5) - autoclosed HOT 1
- github.com/uber-go/tally/v4-v4.1.4: 2 vulnerabilities (highest severity is: 7.5)
- github.com/uber-go/tally/v4-v4.1.5: 2 vulnerabilities (highest severity is: 7.5)
- github.com/uber-go/tally/v4-v4.1.6: 2 vulnerabilities (highest severity is: 7.5)
- [🐛 BUG]: Can't cancel workflow from activity - UnhandledCommand for activity, but workflow is completed HOT 7
- github.com/uber-go/tally/v4-v4.1.7: 2 vulnerabilities (highest severity is: 7.5)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from roadrunner-temporal.