Report on per-thread states (Running, Sleeping, etc)

[ncabatoff]

Report on per-thread states (Running, Sleeping, etc)

[ncabatoff]

I can think of a number of possible explanations. Assuming the error metrics aren't increasing, my best guess is that it's kernel process(s) with an empty cmdline, and hence which are ignored by the '.+' regexp we're matching on by default. Try it with '.*' instead and see if that changes things.

[PeterZaitsev]

Hi,
Looks like same thing. Note In this case I'm running MySQL Benchmark so I kind of expect MySQL to be out there in the top. My theory could be main MySQL thread indeed sleeps most of the time and for this metric other thread statuses are not accounted for, while they are in VMSTAT

[ncabatoff]

If you provide instructions to reproduce I'll have a look. Otherwise:

• try decreasing scrape interval (though you may need to make the config more specific to the mysql procs for process-exporter to keep up)
• try the -threads argument to process-exporter

[PeterZaitsev]

Looking more into it.

Running "stress -c 8" Which creates 8 separate processess spinning on CPU I see:

namedprocess_namegroup_states{groupname="stress",state="Running"} 8

So it works properly for process

In general it does not seems like States are counted per thread even if --threads argument is specified:

namedprocess_namegroup_states{groupname="mysqld",state="Other"} 0
namedprocess_namegroup_states{groupname="mysqld",state="Running"} 0
namedprocess_namegroup_states{groupname="mysqld",state="Sleeping"} 1
namedprocess_namegroup_states{groupname="mysqld",state="Waiting"} 0
namedprocess_namegroup_states{groupname="mysqld",state="Zombie"} 0

We can see only one process "sleeping" here, which I guess is MySQL main thread, we have only one process

namedprocess_namegroup_num_procs{groupname="mysqld"} 1

But many threads:

namedprocess_namegroup_num_threads{groupname="mysqld"} 48

I think for practical purposes "sleeping", "running", "waiting" it is better to count threads thank procs.

[ncabatoff]

I'm sorry, -threads was bad advice, I forgot that I never got around to adding the per-thread state metrics. I will use this issue to track that work and rename it accordingly.

[ncabatoff]

Perhaps I've been thinking about this the wrong way. The 'state' metrics have always been weak because they're gauges: mostly everything is sleeping and waiting, and threads often run only brief intervals, so catching anything in the running state is pretty rare. #62 is probably a better way to provide visibility on how many threads are running or not, via the proposed cpu histograms.

If that takes care of measuring the running threads, we're left with just sleeping and waiting. Sleeping is the default and can be inferred to be the state of what's not running or waiting. For waiting, if I implement #18, we'll know not just how many threads are waiting, we'll see what they're waiting on. In which case this bug possibly becomes a wontfix.

[PeterZaitsev]

Gauges are indeed very volatile but aggregated over period of time they can show good picture.
There is a difference between CPU usage and DiskIO and Waiting. Imagine I have 1 CPU core available. The most I can see is the process using 1 CPU core from CPU time standpoint. However this can come from 1 running thread of from 10 threads in the run queue. It would be best if it would be easy to see the time process/thread spends in Run Queue but not being on CPU or the time process spends waiting on disk IO but I do not think such stats are available and "Running" and "Blocked" Gauges are at least somewhat reasonable estimates.

[ncabatoff]

Much as I hate to break backward compatibility, I'm going to do as you propose in this case, because I've done more testing and it looks like for threaded programs the per-state metrics aren't meaningful. They only report the state of the first thread, which can be very deceptive. The new load-generator cmd I just added always shows up as state S regardless of how many threads are in state D or R.

[ncabatoff]

v0.4.0 pre-release includes this change (state metrics count threads instead of procs.)