I would like to participate in dstat development and responding to issues.

[edi9999]

I would like to participate in dstat development and responding to issues.

[scottchiefbaker]

It was my understanding that the primary reason that dstat was removed from the Fedora repositories was because it lacked Python 3 support. We were never informed of that decision until I read it in the beta release notes. @dagwieers has since added Python 3 support (in about an hour), and probably would have added it sooner if we knew it was a requirement to be included in Fedora.

That said I don't fault you for make a new tool that does all the new fancy things you mentioned above, I do think it's a little sketchy to make a new tool called dstat without at least asking here first. I know that when I hit Fedora 29 and did a yum install dstat and I get a bunch of pcp stuff I was pretty confused.

Dstat hasn't seen a lot of development these days, that is true. Quite honestly that's because it's already very robust and mature, it doesn't need a lot of updates. We could add a bunch of plugins, or clean up the code base sure, but at the end of the day dstat does exactly what it intends to very well without requiring a bunch of maintenance. How much maintenance does top get or ps. Unless there are glaring wholes in functionality I don't think it's a bad thing that a project reaches maturity like dstat has.

[dagwieers]

After all is said and done, there are a few things that don't sound right.

• I don't mind you create a pcp-dstat (like you do for pcp-vmstat, pcp-iostat, etc.), I am all for diversity, but replacing a package and a command with a complete suite is the wrong thing to do. And again it feels like you're using Dstat's popularity to bring in your own suite of tools.
• pcp-dstat is not a drop-in replacement, and won't be for e.g. RHEL8 simply because it is missing plugins. But you only discussed the positives in the proposal, but did not talk about the disadvantages.
• You did not contact the project or its developers, not when you created this fork (a lot of code is still being used), nor when you decided to replace it in the distribution.
• You may say the process was open and discussed, but not informing other shareholders is disingenuous. You also keep repeating Dstat was unmaintained, which is untrue. If you had contacted us with the fact that Dstat would require Python3 support, we would have added it on the spot, like I did yesterday. Because obviously we would like Dstat to ship with newer Fedora and RHEL distributions. (I think you saw this opening as an opportunity for your product, and took advantage of it, not informing us)

And yes, certainly the old, unmaintained version of dstat could be added back into Fedora if its situation changes. There are mechanisms for providing alternative versions of the same named tool, which lets users choose whatever matches their own preferences / needs, and everyone's happy.

I'd like to see you add alternatives to vmstat, iostat, atop, etc. Won't happen.

As I mentioned earlier, dstat is now less code than before

It is not, it is more code. A lot more.

is easier to extend (IMO)

It is easier for you, because you work on PCP. For someone who simply wants to add something to dstat it's actually harder.

as many excellent new features like historical reporting

Which is something that PCP brings, not something that is specific to dstat. Might be useful, but doesn't need a replacement of dstat, would works as well as pcp-dstat or any other name.

[dagwieers]

If it wasn't for Fedora removing dstat from their distribution, I wouldn't have seen this request.
You are now a collaborator. One of three :)

[scottchiefbaker]

Fedora hasn't removed dstat from their respositories as far as I can tell. Is this related to them trying to sunset Python 2.x?

Name         : dstat
Version      : 0.7.3
Release      : 4.fc28
Arch         : noarch
Size         : 888 k
Source       : dstat-0.7.3-4.fc28.src.rpm
Repo         : @System
From repo    : fedora
Summary      : Versatile resource statistics tool
URL          : http://dag.wieers.com/home-made/dstat/
License      : GPLv2
Description  : Dstat is a versatile replacement for vmstat, iostat, netstat and ifstat.
             : Dstat overcomes some of their limitations and adds some extra features,
             : more counters and flexibility. Dstat is handy for monitoring systems
             : during performance tuning tests, benchmarks or troubleshooting.
             : 
             : Dstat allows you to view all of your system resources instantly, you
             : can eg. compare disk usage in combination with interrupts from your
             : IDE controller, or compare the network bandwidth numbers directly
             : with the disk throughput (in the same interval).
             : 
             : Dstat gives you detailed selective information in columns and clearly
             : indicates in what magnitude and unit the output is displayed. Less
             : confusion, less mistakes.

[dagwieers]

It's related to the fact that PCP has a tool that is also named "dstat", and as an argument to replace our tool they give 2 reasons:

• It's dead upstream (last change in 2016)
• It doesn't support Python 3

https://bugzilla.redhat.com/show_bug.cgi?id=1614277

[dagwieers]

It appears that pcp-dstat is a fork of dstat...

[scottchiefbaker]

I'm not a python guru... how complicated would it be to update dstat to support Python 3?

[dagwieers]

I guess it's as easy as doing a diff and then a PR ;-)

[kayb94]

Hi,

obviously nothing has happened so far. Would you allow to me to try to help and invest some time on dstat @dagwieers by making me a collaborator?
I have looked into the dstat replacement of fedora "performanceopilot" aka "pcp", but I find it pretty horrible, to be honest. cloc reports over 200k lines of C code, besides a dozen other languages...

Regards

[dagwieers]

@kayb94 Sure.

PS @edi9999 Never acted on the invite.

[dagwieers]

@scottchiefbaker It is not complicated to make Dstat work on Python 3, it took me less than an hour.

[dagwieers]

Let's close this issue.

[scottchiefbaker]

For what it's worth I also looked at the Fedora "performanceopilot" and it's pretty terrible... it requires a daemon to run? I just want quick and dirty stats, and that's what dstat gets me. I use it everyday.

[natoscott]

Some minor corrections here ... (feel free to ask me anything about PCP and the new dstat there)

| [...] it's pretty terrible

Beauty is in the eye of the beholder I guess, and it depends what you need. PCP is not something directly comparable to dstat - apples, oranges.

| it requires a daemon to run?

No, that's not correct. There is an optional daemon for live data, dstat will use it only if available (it is used for the distributed / remote system analysis aspects) - however, dstat can perform sampling itself still.

FWIW the new dstat shipped in PCP extends the original dstat with historical reporting (i.e. dstat does not only support live data anymore - one can look at yesterdays or last weeks performance now), adds remote system analysis, configuration file support and uses a metric namespace.

Many of these are the more difficult features listed in the original dstat TODO file, and are implemented now because PCP provided APIs for those needs already.

| cloc reports over 200k lines of C code

The dstat in PCP is still all python code, and directly based on the original code - in fact, since PCP provides shared APIs the new dstat involves far less python code than the original dstat.

PCP metrics are often extracted (from PCP archives, from the kernel, and many other domains) using C code though, yes (another original dstat TODO item) - but also via python and other languages.

cheers.

[dagwieers]

@natoscott Why did you decide to call your tool also dstat, ans not pstat or pcpstat ?
I don't understand the reasoning of building your own tool and aiming to replace the original.

[dagwieers]

@natoscott I don't feel good about the fact that replacing dstat with a pcp version has been done completely in the dark. Especially since the pcp version is not a drop-in replacement (according to: https://github.com/performancecopilot/pcp/blob/master/src/pcp/dstat/TODO). The Fedora proposal is very one-sided in the respect.

It seems more like piggybacking on an existing and popular tool to get into distributions.

[dagwieers]

Another problem is that with pcp-dstat you no longer can create your own plugins, you have to add your code to pcp instead. This makes it harder to extend dstat.

And I wonder if you have similar plans to replace atop, collectl, iostat, mpstat, vmstat, etc. ?

[natoscott]

@dagwieers hi there Dag!

| Why did you decide to call your tool also dstat

The new script is actually called pcp-dstat(1), with the initial goal of just being a drop-in replacement. In the end it matched so closely, and was such a quantum leap forward (to an inactive project mind you) that it was decided to phase out the original, unmaintained version - i.e. we were actually requested to completely replace the old dstat, we didn't simply decide it ourselves.

Since there had been no meaningful work on the old dstat code in years, github PRs were being ignored, community developers requesting access to help were being ignored, and we have customers using it ... we were asked to make this as seamless a transition as possible, provide that compatibility symlink for /usr/bin/dstat to pcp-dstat, and remove the old package from RHEL and Fedora.

| [...] done completely in the dark.

I guess it can come across that way now since it wasn't discussed here. But this work was certainly done in the open - on the PCP and Fedora mailing lists, IRC & slack channels, etc - several developers worked on it, all in the open, all open source - and keeping as much of the original dstat code as made sense (note your personal copyrights, --help messages, URL pointers to the dstat website - are all still there in the new version).

In order to make the evolutionary steps the new version has made using PCP (replaying historical data, distributed systems analysis, ...) that work really had to be undertaken by people familiar with PCP. I understand that it may seem to have come out of left field for you, but please also understand there has literally been no activity on the old project for years.

| It seems more like piggybacking on an existing and popular tool

Hmm, that was not the intention. FWIW, PCP is independently popular, has been around longer and was in all distributions long before a new dstat was requested of us - it even runs on other platforms like MacOSX and Windows (where the new dstat now runs too).

It may help to think of this more in terms of "imitation is the most sincere form of flattery" - we wanted to keep dstat alive and moving forward in a sustainable fashion.

We (Fedora, RHEL & others) had problems being caused by the lack of any progress on the original dstat - the python3 situation brought it all to a head, and we soon realised we could provide some interesting new functionality by bringing PCP and dstat together.

| Another problem is that with pcp-dstat you no longer can create your own plugins

Plugins are now (text) files below /etc/pcp/dstat and ~/.pcp/dstat - they can use any of the thousands of metrics made available by PCP (well, millions I guess, since PCP accepts the increasingly popular Prometheus exposition format for metrics too).

| This makes it harder to extend dstat.

On the contrary, it is far easier to create simple text-based configuration files than to write new python code...

$ cat /etc/pcp/dstat/fs
#
# pcp-dstat(1) configuration file - see pcp-dstat(5)
#

[fs]
width = 6
label = filesystem
files = vfs.files.count
inodes = vfs.inodes.count - vfs.inodes.free

Gives traditional dstat output like:

$ pcp dstat --fs 10 2
--filesystem-
files  inodes
17280    162k
17280    162k

Text plugins provide a separation of the sampling of metric values from the way they are displayed, which is all quite mixed up together in the old dstat. This neat separation is what enables historical operation, since metrics are referenced by name and can be retrieved from archives:

$ pcp --archive 20190109 --start @10:00am dstat --time --fs 10sec 2
----system---- --filesystem-
     time     |files  inodes
09-01 10:00:00|11744    419k
09-01 10:00:10|12018    423k

or across the network:

$ pcp --host xxx.redhat.com dstat --time --fs 10 2
----system---- --filesystem-
     time     |files  inodes
17-01 15:10:56|14304    132k
17-01 15:11:06|14304    132k

Finally, please don't take any of what I've said personally, or to be anti-dstat, or antagonistic in any way - that's certainly not my intention here. It was very refreshing to see that dstat, like PCP (but unlike many tools), embraces the fact that metrics have units! - this was one of the reasons the integration with PCP works so seamlessly, and enabled us to achieve the things we have in this new version. Thank you!

cheers.

[dagwieers]

On the contrary, it is far easier to create simple text-based configuration files than to write new python code...

That's quite untrue, this only works if the data was already obtained by PCP in some way. The proof is that a lot of existing plugins are not supported in pcp-dstat. You can't just add a text file to get those counters in pcp-dstat, you need to extend PCP.

[scottchiefbaker]

I'm not a developer of dstat (not really) because my Python skills are severely lacking. I'm just a very avid fan of dstat. I'm a system administrator at an ISP and I use dstat every day to monitor the health of my servers. My voice probably holds very little weight in this conversation (compared to @dagwieers) but to put in my two cents here.

If the primary reason that dstat was removed from Fedora original was due to lack of Python 3 support (which we were not made aware of) then when we do release dstat with Python 3 support (hopefully soon) the original dstat tool be reinstated and pcp-dstat be renamed to something else.

[dagwieers]

@scottchiefbaker Of course not. The Fedora team has been played, they were promised a drop-in replacement with only additional benefits, but instead people get the PCP suite installed for free and some plugins won't work (i.e. the top-plugins are missing).

[scottchiefbaker]

For the record, I do think PCP stuff is cool, and definitely has it's uses.

Personally I enjoy the simplicity and ease of use of dstat. I'm not trashing PCP, or even pcp-dstat, there is room for both. Preferably not with the same conflicting and confusing name though.

[natoscott]

@dagwieers @scottchiefbaker good morning!

Hmm, this work on a new dstat seems to have invoked some emotional responses - I apologise unreservedly, and state again - not my intention here. I, like you Scott (and surely Dag too), am a fan of dstat and have enjoyed adding these new features to a very solid tool. The old dstat certainly lives on in the new dstat if you look through the code.

| | The Fedora team has been played, they were promised [...]

Sorry, no. Noone was tricked, and there was no such promise. I am part of 'the Fedora team', FWIW - there were completely open discussions and there was literally not a single word in favour of the old dstat versus the new, over several months of chat and development - on the contrary, plenty of folk were keenly awaiting the new features. Anyway, I do hope we can move forward and at least discuss the technical side of things (as we're doing).

FWIW Scott - that document was circulated far and wide long before f29 and describes in detail the reasons the old dstat was being updated with the new dstat, and it describes more than just the issues around lack of python3 support (I also outlined these issues earlier in this thread, too).

And yes, certainly the old, unmaintained version of dstat could be added back into Fedora if its situation changes. There are mechanisms for providing alternative versions of the same named tool, which lets users choose whatever matches their own preferences / needs, and everyone's happy.

| | [...] it is far easier to create simple text-based configuration files than to write new python code...
| That's quite untrue,

Setting that aside for the moment, in terms of the technology - what are your thoughts on using named metrics and configuration files for the report formatting aspects of plugins (as in the new dstat), instead of python code that mixes both sampling and presentation of the data (the old dstat way)?

| | You can't just add a text file to get those counters in pcp-dstat, you need to extend PCP.

That's correct, yes - but there's two separate things we're talking about here which is clouding the discussion. In the old dstat, a plugin was a combination of metric extraction logic and logic around how those metrics are to be presented. In the new dstat, a plugin uses existing metrics PCP makes available (which, like in the old dstat, can involve a user adding their own metrics via python, C, perl, ... code).

| | this only works if the data was already obtained by PCP in some way.

Totally. But this seems to me a circular argument - we can extend PCP with code to sample new metrics just like we could extend the old dstat with new code to sample new metrics. And we can extend the new dstat with configuration files describing how to layout reports just like we could extend the old dstat in a similar fashion (albeit via code, instead of simple configuration files).

With named metrics and sampling of metrics done below an API, many tools get to use those metrics. Whereas in the old dstat, there's no ability to share code between tools of course. This results in a larger available metrics pool, and the presence of an API leads to other types of tools as well, e.g. for

• strip charting
• cockpit dashboards
• evaluating performance rules
• all sorts of metrics exporters ... pcp2zabbix, pcp2spark, pcp2graphite, pcp2influxdb, etc ... above and beyond the basic export-to-csv dstat builds in, etc.

| | [...] a lot of existing plugins are not supported in pcp-dstat.

All the important kernel and other metrics are covered, and we have many new reporting "plugins" and far more metrics available to choose from. There are a few obscure (e.g. IPMI) metrics which we left (asked, but found noone using them - easy to add if needed though), and the top-like plugins as you mentioned Dag. We'll add support for the 'top' concept in due course, and in a generic way so that any plugin can use that concept (via config file).

Personally I enjoy the simplicity and ease of use of dstat.

Agreed - its a neat tool, and I hope people would at least try out the new version before bashing it.

As I mentioned earlier, dstat is now less code than before, is easier to extend (IMO), and has many excellent new features like historical reporting ... a neat evolution on the old dstat, and next-generation alternative for folk looking for something more.

cheers.

[natoscott]

@dagwieers hi!

After all is said and done, there are a few things that don't sound right. [...]
(I think you saw this opening as an opportunity for your product [...]
You also keep repeating Dstat was unmaintained [...]

I think we're going round in circles on these topics Dag, and are unlikely to change each others thinking.

Let me just end by saying if I thought there was value in doing so I could have pointed you toward our python3 code - we largely did the same python3 port when building the new dstat of course. However, reading bug reports like this one and others like it, and all the unacknowledged dstat pull requests, I reached the conclusion there was no point.

Anyway, lets discuss the technical aspects and features of the code, perhaps we can find common ground there. I asked you earlier, but got no response ...

| what are your thoughts on using named metrics and configuration files
| for the report formatting aspects of plugins (as in the new dstat), instead
| of python code that mixes both sampling and presentation of the data
| (the old dstat way)?

Any thoughts? I see in your old TODO file you had items like:

• Create a nicer interface for plugins (with meaningful names, eg. not nick)
• Design proper object model and namespace for all possible stats

I think the above items are solved now in the new dstat, but I may have misunderstood your original notes.

As I mentioned earlier, dstat is now less code than before

To clarify, that comment I made was based on this approximation ...

$ wc -l dstat/dstat
2856 dstat/dstat
$ wc -l pcp/src/pcp/dstat/pcp-dstat.py
1653 pcp/src/pcp/dstat/pcp-dstat.py

is easier to extend (IMO)

By that I meant it is now easier for users to extend dstat with new reports, since the barrier to entry is not 'write python code' - it is 'edit a configuration file' (as @scottchiefbaker states, many users are not python programmers). To add sampling of new metrics involves new code for both old dstat and PCP, naturally.

as many excellent new features like historical reporting

Which is something that PCP brings, not something that is specific to dstat.

I'm not following the point you're making there. Almost all performance tools provide historical analysis capabilities - perf, collectl, atop, sar - all recognise that this is a crucial part of the performance analysis process, and it was much needed functionality in dstat that is now provided.

Have a great weekend!

cheers.

[trinitronx]

@natoscott :

However, reading bug reports like this one and others like it, and all the unacknowledged dstat pull requests, I reached the conclusion there was no point.

This seems like perhaps there is a bit of Anchoring effect here? (Could have also been a bit of confirmation bias, status-quo bias, and/or system justification bias sprinkled in?... ^{a dash here... a dash there 🤪 🤣}). _{Note: nothing intended here other than to gather attention to the hidden subconscious and unconscious forces that affect all human beings.}

While it may be true that mature projects, (especially those with fewer maintainers) have less overall activity than newer ones still in heavy development, basing one's decision to not communicate or collaborate on the current state of pull requests and issues, while leaning towards a preconceived notion that there would be total inaction or futile efforts, at first glance seems to be a bit of a logical fallacy or unconscious cognitive bias. One might never know what response you might get unless you try. Case in point: After reaching out to this project, you did get a response (abeit not as positive of an outcome as one might hope had it been earlier in the situation).

This seems to be an unnecessary outcome: that a newer project or even a ported fork can't just choose a new name. There are many cases in the FOSS community of this happening... just look at how many forks of projects with suffix "ng" there are out there 😄

I think the main point @dagwieers was getting at here was that it would have been nice to see at least some attempt at reaching out earlier to the community here when the original need to migrate towards Python3 came up. A lot of python projects have rather trivial migration paths (e.g.: simply run 2to3, and a linter.. fix the errors / warnings, etc... ). I think that early-on communication would have gone a long way towards creating a collaborative environment and keeping the two projects communities amicable with each other, and avoided unnecessary dropping of such a great tool by the original developer.

I realize there are some hurt feelings here, and diverging interests. I also appreciate the sense that there are big enterprises and forces out there that actually do eat up and kill off projects (due to over-competitive fallout of capitalism)... RedHat included! (See: CoreOS EoL).

I guess it can come across that way now since it wasn't discussed here. But this work was certainly done in the open - on the PCP and Fedora mailing lists, IRC & slack channels, etc - several developers worked on it, all in the open, all open source - and keeping as much of the original dstat code as made sense (note your personal copyrights, --help messages, URL pointers to the dstat website - are all still there in the new version).

I'm a small contributor to this project, and still appreciate and use the original dstat in daily work. Yet I'm just learning and hearing about this series of unfortunate events leading to the */bin/dstat CLI namespace collision now causing a complete abandonment of this project. 😢

Sorry we didn't notice? 🤨 This is all very strangely reminiscent of the 1st and 3rd Chapters of Douglas Adams' classic "The Hitchhiker's Guide to the Galaxy" where Arthur Dent's home was being demolished. Yet, the plans for the bypass expressway were "on display" _{(in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard.")}, and then later the Vogons came to Earth announcing it's imminent demolition required for building of a hyperspatial express route through our star system _{(but the plans were "on display" at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now. ... What do you mean you’ve never been to Alpha Centauri? Oh, for heaven’s sake, mankind, it’s only four light years away, you know. I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. Energize the demolition beams! 💣💥🤯🤪)}.

In order to make the evolutionary steps the new version has made using PCP (replaying historical data, distributed systems analysis, ...) that work really had to be undertaken by people familiar with PCP. I understand that it may seem to have come out of left field for you, but please also understand there has literally been no activity on the old project for years.

It's great that you're innovating and even imitating dstat, which I see as a compliment ("imitation is the most sincere form of flattery"). However, this new pcp tool sounds like it addresses a lot of separate concerns than the original dstat that parts of it were based on. It's great to see a new option for metrics out there with support for newer interfaces such as Prometheus scrape API. I once thought it would be nice if dstat could export data to Graphite or some TSDB, especially with some of the more difficult stats to gather with other tools using C plugins and text-based DSL config files, collectd plugins and the like. This required digging through loads of documentation, some of which was difficult to find for lesser known stats & third-party plugins. All arguments aside (_{really this shouldn't have needed to be a competition between tools}), there does seem to be a bit of Pro-innovation bias, and divergent concerns going on here. Yet, does it really need to be a complete replacement of dstat in the Linux command namespace? (Yes, I'm aware of /etc/alternatives, which is fine to use in some circumstances, however if the intent was to simply fork and maintain a separate project bundled with other tools then why create an unnecessary Conflicts package relationship?)

Anyway, it's sad to hear yet another casualty of RedHat has occurred (again.. see CoreOS EoL). I just hope it's not too late to save a great project and keep it going in the original spirit of the FOSS community.