simonnagl/netdata-java-orchestrator

Netdata Java Plugin Daemon

netdata-java-orchestrator's Issues

Auto detection of local JAVA processes.

The JMX-Plugin should be able to auto-detect and connect to local running java processes.

This needs some research:

This is probably the solution: http://www.javaworld.com/article/2073919/core-java/groovy--jmx--and-the-attach-api.html

Put every JMX server in it's own dashboard section.

This requires to add family option to configuration.

How to remove the Demo Java Chart?

Hi . I want to remove the jmx NetdataJavaDaemon chart. How to do that?
@simonnagl

Prevent a bash process in the netdata process tree.

Use exec java ... to run java to prevent a bash process in the netdata process tree

author: @ktsaou

Inconsistent use of chart properties 'family' and 'context'

Clone of netdata/netdata#3632

For description and discussion please use the clone.

Plans to implement jvm process heap monitoring?

Is there a way to monitor jvm process heap with this plugin? If not this feature could be very useful. Great plugin btw!

Add system test

It would be appropriate to have a test which runs the compiled char with some example configuration and tests the output of the plugin.

Optimize data collection.

The JMX API supports to query multiple attributes of one MBean with one request. We should support that too.

Needed research:

• Does this improve performance?
• Is using getAttributes slower if only fetching one attribute?

Needed changes:

• Extend MBeanQueryInfo: MBeanQueryInfo should contain information about the MBean and a list of Attributes to collect and where to store them.
• Change initialization of MBeanServerCollector to fit.
• Change data collection of MBeanServerColletor to fit.

Add CPU usage per thread

from: java.lang:type=Threading

1. Find thread IDs: AllThreadIds
2. (find names to ids?)
3. Query ThreadCpuTime and maybe ThreadUserTime for every thread.

We need to find a way to configure this type of collection.

Add divider multiplier option to JmxDimensionConfiguration

Discussion: Which configuration language to use.

@ktsaou started a discussion about configuration languages in #39. Let's continue it here.

Currently we use json. Me may switch this.

Arguments of @ktsaou:

• standard practice for java apps config files is xml.
• json does not support comments so extensive documentation is needed in separate files.
• For node.js plugins I would definitely use yaml if there was a javascript-only implementation yaml library.

My Arguments:

• XML is not very readable.
• We extend JSON to support Java/C-like comments in this project
• Jackson (the parser library we use) has a yaml extension. I never used it but we could try.

Mark charts of dead Java processes obsolete.

If a Java process gets killed we currently do not collect values any more, but the chart get's empty. For auto-configured processes, if the process is started with a new PID there is no chance to attach this process to the old charts.

We do not want to spam the metadata dashboard creating new charts and not removing the old ones. We want to mark charts which we won't refresh any more as obsolete.

Add connection timeout when making JMX connections.

Otherwise the initialization process may hang for a very long time.

https://community.oracle.com/blogs/emcmanus/2007/05/23/making-jmx-connection-timeout

How to get Text value from jmx

eg: jvm version , I see the comment, only number attribute is supported

How to get alarms

Hi,
I am using Netdata 1.17.0 and Java Version 1.8
and i installed this "netdata-java-orchestrator" using the script "netdata-java-orchestrator-installer.sh" as mentioned in "README.md" and everything is successfully executed and i got "JMX-CHARTS" in the Netdata but i didn't the "Alarms". If there is any procedure to configure Alarms please help me out

@simonnagl

Add license header to every file.

Every source file should contain this header.

/*
 * Copyright (C) 2017 Simon Nagl
 *
 * metadata-plugin-java-daemon is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 *
 */

Publish first version to a Maven repo

Edit: I am not sure this brings any benefit to this project...

Integrate this into netdata.

Author: @ktsaou

Integrating this into netdata would also be nice. I guess this would require some more work to build it using the netdata automake system and install it with the rest.

What do you mean with integrating?
Putting it completely into the netdata project? When I started implementing this I decided to do it in a separate project mainly because java has a much different build system. My thoughts where these: Netdata could add this to the installer. The installer will decide if java is there. If yes it downloads this plugin and runs the plugins installer. If no we do not need this plugin. We won't be able to compile it anyways.

Plugin does not compile due to offine mode

Hi there,

in line 225 of netdata-java-orchestrator-installer.sh, Maven is configured to run in offline mode. As a result, the dependencies cannot be resolved properly and the installation fails. Removing the -o switch solves the problem.

Cheers,
marcus

Combined Charts

We may support charts with combined information of all monitored JMX processes.

For example we could provide a chart CPU which displays CPU usage of java processes with one dimension per process.

Process Cpu Load

Add a process CPU Load chart.

Collect value ProcessCpuLoad from java.lang:type=OperatingSystem

Recommendation:

• Ability to collect double values

Automate installation

Installation should be easy. Only one command should be needed to set everything up.

Add health configuration files.

For the charts of the default configuration, add health configuration.

https://learn.netdata.cloud/docs/agent/health/reference/

Composite Data Support

Add support for MBeans returning CompositeData.

Cassandra DB/Cluster monitoring

Clone of netdata/netdata#3101

Documentation:
https://github.com/ezbz/jmxtrans-lib/blob/master/cassandra/output/mycass01-cassandra.json

Support jmx authentication

Clones netdata/netdata#3046

Currently the JMX module of the java plugin can only connect to JMX Servers without authentication. We should provide a way to configure authentication for custom servers.

This request did come from @h0lzi.

Process Cpu Percent

Add a process CPU percent chart.

Value ProcessCpuTime from java.lang:type=OperatingSystem is in nanoseconds.

Recommendations:

• #7 Add multiplier/divider/factor to JmxDimensionConfiguration

Unable to pull minecraft server MBean data

I'm trying to use this to pull MBean information from my minecraft server(s) to monitor internal tick times.

My first issue is that if I define the server it won't find it, however if I leave the "jmxServers" list empty it will find the connection regardless if "autoDetectLocalVirtualMachines" is set to true or false. If I define a jmxServers list entry, it doesn't show any information at all.

For right now I'm testing it with an empty list and autoDetect set to true.

Here's the jmxServer entry that I have tried. This server is started with -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=42001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.1.137

        {
            "name": "Minecraft1",
            "serviceUrl": "service:jmx:rmi:///jndi/rmi://127.0.0.1:42001/jmxrmi"
        }

I have tested the connection locally with jmxterm to ensure it works and the MBean is available:

$>open 127.0.0.1:42001
#Connection to 127.0.0.1:42001 is opened
$>get -b net.minecraft.server:type=Server averageTickTime
#mbean = net.minecraft.server:type=Server:
averageTickTime = 1.5295846;

I've also tested this with VisualVM over the network and am able to get this MBean data.

I've gone and thrown in some extra logging in a few places just to verify what information it's passing around but it just confirms that it's not able to find it.

2021-08-18 10:12:35: java.d: WARNING: org.firehol.netdata.module.jmx: dimension objectName = net.minecraft.server:type=Server    
2021-08-18 10:12:35: java.d: WARNING: org.firehol.netdata.module.jmx: getMBeanQueryForName has been called, object = net.minecraft.server:type=Server, attr = averageTickTime  
2021-08-18 10:12:35: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: [org.firehol.netdata.module.jmx.exception.JmxMBeanServerQueryException] Could not query attribute 'averageTickTime
' of MBean 'net.minecraft.server:type=Server' ... javax.management.InstanceNotFoundException: net.minecraft.server:type=Server ... net.minecraft.server:type=Server Detail: net.minecraft.server:type=Server

Is there a reason you know of why it wouldn't be able to find the MBean/attribute that I'm able to pull using other methods? I'm still trying to look through java and see, but I'm not super well versed in java.

Recheck for new or restarted JMX Servers

@h0lzi wrote at netdata/netdata#364:

One thing I notice. If I restart the tomcat the plugin will not reconnect.
I'm not sure what the best way to handel that is. Maybe if there was a connection try every x seconds for y minutes.
(Make that a config option?). Off cause it needs to bail out eventually.
But besides that basic stuff seems to work for me.

He is right. We need an option to periodically recheck if the configured JMX servers are there and rerun auto detection if it is on.

Migrated from netdata/netdata#3057

Use Dependency Inversion Principle (DIP) to make services testable.

We need to decouple the dependencies of JmxPlugin and other singletons to make them testable.
We might use Guice to inject the dependencies at runtime or maybe it is enogh for now to initialize them before the program runs.

Uptime

Add Uptime Chart of the JVM

Value Uptime of java.lang:type=Runtime returns milliseconds.

Recommendations:
#7 Add divider/multiplier/factor option to JmxDimensionConfiguration

Rename the project

The project name is misleading. The program is intended to run as a netdata plugin.
netdata will run the process. This has nothing to do with a daemon process.

Also I am not sure if I should keep this project open for other java collectors (currently there is only an JMX collector).

If yes the new name will be netdata-jmx-plugin
If not the new name would netdata-java-plugin

@ktsaou: Do you want me to help making this decision? I am asking because I wanted to ask you would accept moving this project to the netdata organization. I am asking you know because the naming, and with it the decision closing this project to other java collectors/plugins might affect your decision.
If you do not want to have anything to do with this please just ignore my question.

BufferPool Chart

Add a chart with the values of java.nio:type=BufferPool beans.

Recherche is needed what the values represent exactly.

Installer: Do not overwrite modified configuration files.

Currently the installer always copies the newest configuration files to the configuration directory. The installer should not overwrite existing configuration files.

Move formatter configuration to jar

After #24 the configuration property java.util.logging.SimpleFormatter.format will be in the installer. I think it should go into the source code or the resources of the jar file.

Memory is not support:

I have add some mem metrics, I got some error:

        {
        // Unique ID of this chart in this list.
        "id": "server",
        "title": "session",
        // Unit of the collected value
        "units": "",
        // orders charts. Lower numbers make the charts appear before the ones with higher numbers
        "priority": "8000",
        // Possible chart types: AREA, LINE, STACKED
        "chartType": "AREA",
        // Possible dimension algorithms:
        // ABSOLUTE, INCREMENTAL, PERCENTAGE_OF_ABSOLUTE_ROW, PERCENTAGE_OF_INCREMENTAL_ROW
        "dimensionAlgorithm": "ABSOLUTE",
        // List of dimensions to add to the chart.
        "dimensions": [
            {   
                "from": "Catalina:type=Manager,host=localhost,context=/",
                "value": "activeSessions",
                "name": "activeSession",
                "multiplier": "1",
                "divisor": "1",
                "hidden": "false"
            },
            {   
                "from": "Catalina:type=Manager,host=localhost,context=/",
                "value": "maxActive",
                "name": "maxActive",
                "multiplier": "1",
                "divisor": "1",
                "hidden": "false"
            },
            {
                "from": "Catalina:type=Manager,host=localhost,context=/",
                "value": "maxActiveSessions",
                "name": "maxActiveSessions",
                "multiplier": "1",
                "divisor": "1",
                "hidden": "false"
            },
            {
                "from": "Catalina:type=Manager,host=localhost,context=/",
                "value": "essionCounter",
                "name": "sessionCounter",
                "multiplier": "1",
                "divisor": "1",
                "hidden": "false"
            },
            {
                "from": "Catalina:type=Manager,host=localhost,context=/",
                "value": "rejectedSessions",
                "name": "rejectedSessions",
                "multiplier": "1",
                "divisor": "1",
                "hidden": "false"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "HeapMemoryUsage.committed",
                "name": "HeapMemoryUsagecommitted"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "HeapMemoryUsage.max",
                "name": "HeapMemoryUsagemax"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "HeapMemoryUsage.used",
                "name": "HeapMemoryUsageused"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "NonHeapMemoryUsage.committed",
                "name": "NonHeapMemoryUsagecommitted"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "NonHeapMemoryUsage.max",
                "name": "NonHeapMemoryUsagemax"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "NonHeapMemoryUsage.used",
                "name": "NonHeapMemoryUsageused"
            },
            {
                "from": "java.lang:type=Memory",
                "value": "ObjectPendingFinalizationCoun",
                "name": "ObjectPendingFinalizationCoun"
            }
        ]
    },

erro log:
rehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.committed' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.committed
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.max' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.max
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.used' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.used
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.committed' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.committed
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.max' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.max
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.used' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.used
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'ObjectPendingFinalizationCoun' of MBean 'java.lang:type=Memory' Detail: No such attribute: ObjectPendingFinalizationCoun
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'activeSessions' of MBean 'Catalina:type=Manager,host=localhost,context=/' Detail: Catalina:type=Manager,host=localhost,context=/
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'maxActive' of MBean 'Catalina:type=Manager,host=localhost,context=/' Detail: Catalina:type=Manager,host=localhost,context=/
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'maxActiveSessions' of MBean 'Catalina:type=Manager,host=localhost,context=/' Detail: Catalina:type=Manager,host=localhost,context=/
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'essionCounter' of MBean 'Catalina:type=Manager,host=localhost,context=/' Detail: Catalina:type=Manager,host=localhost,context=/
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'rejectedSessions' of MBean 'Catalina:type=Manager,host=localhost,context=/' Detail: Catalina:type=Manager,host=localhost,context=/
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.committed' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.committed
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.max' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.max
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'HeapMemoryUsage.used' of MBean 'java.lang:type=Memory' Detail: No such attribute: HeapMemoryUsage.used
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.committed' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.committed
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.max' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.max
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'NonHeapMemoryUsage.used' of MBean 'java.lang:type=Memory' Detail: No such attribute: NonHeapMemoryUsage.used
2018-01-31 01:31:02: java.d: WARNING: org.firehol.netdata.module.jmx: Could not query one dimension. Skipping... Reason: Could not query attribute 'ObjectPendingFinalizationCoun' of MBean 'java.lang:type=Memory' Detail: No such attribute: ObjectPendingFinalizationCoun

Enhance Chart ordering

I decided to do this as simple as possible. We just add priority to configuration and let metadata do the ordering.

Memory Usage

Add Memory usage Charts

Dependes on:
#6 Composite Data Support

Configuration option to disable JMX plugin.

Support monitoring array size of MBean attributes

java.lang:type=Threading DeadlockedThreads. Display number of DeadlockedThreads.

DeadlockedThreads is a long[]. We want to count the size of long[]. We need to find a way to configure this kind of collection.

Documentation

Add documentation for:

• Installation
• Configuration of the daemon.
• Configuration of JMX plugin.