Comments (11)
Looking at BaseRTMPTConnection class: it uses LinkedBlockingQueue for outgoing messages and attemps to handle queue overflow gracefully by discarding messages. The problem is that LinkedBlockingQueue is created with default capacity, which is Integer.MAX_VALUE (2'147'483'647). I suppose this is too big and needs to be changed to a value that makes sense for Red5 default heap size of 768M.
from red5-server.
Sergey, thanks for the report and the plumbr details. I'll look at the latest code and see if the default size is still be used.
from red5-server.
For reference the class is now located in red5-server-common https://github.com/Red5/red5-server-common/blob/master/src/main/java/org/red5/server/net/rtmpt/BaseRTMPTConnection.java
from red5-server.
Fixed at commit Red5/red5-server-common@498780c
from red5-server.
Thanks a lot for quick response! Will this fix be included in then next Red5 release? Is there an approximate date for the next release?
from red5-server.
Yes, it'll be in the next release 1.0.4; which we are trying to get out before 2015.
from red5-server.
Great! Thanks again!
from red5-server.
We've configured red5 to leave memory dumps after OOM, here are the dumps that we got:
1 - https://drive.google.com/file/d/0B1vFExHovKwobllpNEt3Zk5RNTg/view?usp=sharing
2 - https://drive.google.com/file/d/0B1vFExHovKwoMmdaaEREYzJSVk0/view?usp=sharing
3 - https://drive.google.com/file/d/0B1vFExHovKwocTRwU015aVA5ZU0/view?usp=sharing
4 - https://drive.google.com/file/d/0B1vFExHovKwoZm1ubjlGb2pESW8/view?usp=sharing
5 - https://drive.google.com/file/d/0B1vFExHovKwobmNHSUp4TEs4V28/view?usp=sharing
After analyzing the dumps with jvisualvm the following interesting facts we found:
a) In each dump the number of RTMPTConnections was larger then we expected, according to our test setup we expected 6 connection in each dump, but the number of connections was:
dump 1 - 243,
dump 2 - 12,
dump 3 - 24,
dump 4 - 18,
dump 5 - 24.
b) In each dump the number of RTMPTConnections with small (< 200) amount of pending outgoing messages was about 6:
dump 1 - 9,
dump 2 - 7,
dump 3 - 7,
dump 4 - 7,
dump 5 - 7.
(the following OQL query in jvisualvm was used:
'select c from org.red5.server.net.rtmpt.RTMPTConnection c where c.pendingOutMessages.count.value < 200' )
a) and b) together support a hypothesis, that in our case memory leak happens due to unclosed RTMPT connections.
This raises 3 questions:
- Where should we continue investigation to figure out, what is causing unclosed connections?
- Does it makes sense for red5 to leave memory dumps on OOM by default? Would you accept a patch for it?
- Does it makes sense for red5 to expose monitoring of RTMPT connections via JMX (similarily to monitoring of streams that we have now)? Would you accept a patch for it?
Thanks in advance!
from red5-server.
Did you use the patched / latest server build?
On Wed Dec 24 2014 at 12:00:55 PM Sergey Smyshlyaev <
[email protected]> wrote:
We've configured red5 to leave memory dumps after OOM, here are the dumps
that we got:1 -
https://drive.google.com/file/d/0B1vFExHovKwobllpNEt3Zk5RNTg/view?usp=sharing
2 -
https://drive.google.com/file/d/0B1vFExHovKwoMmdaaEREYzJSVk0/view?usp=sharing
3 -
https://drive.google.com/file/d/0B1vFExHovKwocTRwU015aVA5ZU0/view?usp=sharing
4 -
https://drive.google.com/file/d/0B1vFExHovKwoZm1ubjlGb2pESW8/view?usp=sharing
5 -
https://drive.google.com/file/d/0B1vFExHovKwobmNHSUp4TEs4V28/view?usp=sharingAfter analyzing the dumps with jvisualvm the following interesting facts
we found:
a) In each dump the number of RTMPTConnections was larger then we
expected, according to our test setup we expected 6 connection in each
dump, but the number of connections was:
dump 1 - 243,
dump 2 - 12,
dump 3 - 24,
dump 4 - 18,
dump 5 - 24.
b) In each dump the number of RTMPTConnections with small (< 200) amount
of pending outgoing messages was about 6:
dump 1 - 9,
dump 2 - 7,
dump 3 - 7,
dump 4 - 7,
dump 5 - 7.
(the following OQL query in jvisualvm was used:
'select c from org.red5.server.net.rtmpt.RTMPTConnection c where
c.pendingOutMessages.count.value < 200' )a) and b) together support a hypothesis, that in our case memory leak
happens due to unclosed RTMPT connections.This raises 3 questions:
Where should we continue investigation to figure out, what is causing
unclosed connections?Does it makes sense for red5 to leave memory dumps on OOM by default?
Would you accept a patch for it?Does it makes sense for red5 to expose monitoring of RTMPT connections
via JMX (similarily to monitoring of streams that we have now)? Would you
accept a patch for it?Thanks in advance!
—
Reply to this email directly or view it on GitHub
#40 (comment).
from red5-server.
As for creating dumps automatically, I'd have to say no as it would cause confusion to the majority of users. I would however be glad to post a wiki page about it, if you want to write one up.
from red5-server.
Regarding server version, we are using 1.0.1. Was there any specific fix on RTMPT connections handling after 1.0.1? Anyway, I will try to reproduce the problem on a more recent version. What version would you recommend: 1.0.3 or 1.0.4-SNAPSHOT?
Regarding memory dumps wiki page, I'll write one when I have time.
What about monitoring RTMPT connections via JMX? I think it would be a valuable feature.
from red5-server.
Related Issues (20)
- red5 server native dependencies HOT 4
- Unrecognized VM option 'UseZGC' HOT 7
- can't running test with Junit and maven HOT 2
- 打包后运行 HOT 2
- 运行一会儿闪退
- Red5 on Cloudron HOT 1
- http-flv support HOT 6
- This is question for use red5. It’s not issues HOT 4
- Error upon starting the server HOT 2
- Red5 server not running on Linux ubuntu 20.04 HOT 1
- i find a bug in red5-server HOT 2
- o.r.s.stream.consumer.File Consumer - Queue size is greater than threshold.
- org.red5.server.stream.StreamService - Bad name HOT 3
- "The JAVA_HOME environment variable is not defined correctly" HOT 8
- stream keys are immediately burned HOT 3
- Custom Playback directory not working HOT 1
- Video not streaming
- Official Docker image for Red5 Community version? HOT 4
- Red5 server wont start. bootstrap class issue. unable to find working solution
- CORS error on react.js HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from red5-server.