omergertel/pyformance

Performance metrics, based on Coda Hale's Yammer metrics

pyformance's Issues

Add basic logging for easier debugging

Would be beneficial for debugging if the module did some logging. For example, when Reporter starts for the first time or flushes to sink later on. Just the normal, basic stuff.

New release

Hi,

Would you consider making a new release soon?

Many thanks,

David

How can we extend this library to include a reporter to openTSDB

I want to use the metrics and MetricsRegistry from this library and push the metrics to openTSDB. However, the metric only has a key which is not able to support tags from openTSDB. In that case, how can we create an OpenTSDBReporter which is similar to HostedGraphiteReporter for graphite? Thanks.

Requesting new release to pip

Hi,

Recently there were some fixes for the InfluxDB reporter added to Pyformance. However, these changes don't seem to be present in the version of Pyformance available via pip (0.3.4). I was hoping you could do a new release so we wouldn't have to install the latest code from Github.

Thanks!

Fork of pyformance

Hello,

I think pyformance is great and I use it for some projects.
Therefore I fixed some stuff and added missing features.

The fork is available at:
https://github.com/hajs/pyformance/
The modifications are documented in the file CHANGES.

I would be glad if you consider merging my code.

CsvReporter does not close file handles after finishing

There's currently no way to close the file handles opened by CsvReporter. This issue is demonstrated when running unit tests in a Windows environment:

python  "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.3.2\helpers\pycharm\utrunner.py" C:\dev\git\pyformance\tests\test__csv_reporter.py true

Testing started at 11:50 PM ...

Error
Traceback (most recent call last):
  File "C:\dev\git\pyformance\tests\test__csv_reporter.py", line 27, in tearDown
    shutil.rmtree(self.path)
  File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\-\\AppData\\Local\\Temp\\tmpd295pp0o\\gauge1.csv'

Process finished with exit code 0

point tags in pyformance?

Hi, thanks for this library.

Does pyformance support point tags? For example, AFAIK a graphite metric is a name, a timestamp (possibly implied), and a value, e.g. "cpu.idle [now] 42". Some metrics monitoring solutions allow point tags, for example you could have:

cpu.idle 42 host=myFoo env=prod
cpu.idle 23 host=myBar env=dev

and on the solution you could do aggregations by environment, or host, etc.

If pyformance does not support point tags, would allow a contribution adding support for them? Along with a reporter that knows how to stream points to a TCP socket?

Support "arbitrary" mean throughput intervals in Meter

The class ExpWeightedMovingAvg has a parameter which allows you to specify the period of reporting, yet there are 3 hardcoded instances in Meter:

            self.m1rate = ExpWeightedMovingAvg(period=1, clock=self.clock)
            self.m5rate = ExpWeightedMovingAvg(period=5, clock=self.clock)
            self.m15rate = ExpWeightedMovingAvg(period=15, clock=self.clock)

Which results in only getting those metrics in the dashboard.

Would be great if you could also specify your own intervals. Something like Meter(clock=self._clock, [1, 5, 15, 30, 60])

Would this be useful? Seems fairly trivial to implement, would be happy to create the PR.

Upload to Conda Forge

Is it possible this package could be added to conda forge? Although it is already on pip it would be great if conda users could also easily install pyformance!

influxdb reporter is not pushing data

Hi,

I just wrote simple script to get me started on pyformance, seems like influxdb reporter is not pushing any data to influxdb, my script is as below, can someone let me know if something is wrong with it?

from pyformance.registry import MetricsRegistry
from pyformance.reporters.influx import InfluxReporter
import time

registry = MetricsRegistry()
counter=registry.counter("test")

reporter=InfluxReporter(registry=registry)
reporter.start()

while True:
    time.sleep(2)
    counter.inc()
    print counter.get_count()

No lock on Counter.dec()

It seems like you forgot to lock the Counter on dec():

pyformance/pyformance/meters/counter.py

Line 20 in b71056e

def dec(self, val=1):

sanic multi worker metrics are not populated

I am trying to add metrics to sanic application.
When running 1 worker everything looks fine, but when configuring more workers no metrics being sent to carbon.

import random
import time

from sanic import Sanic
from sanic.response import json

from pyformance import MetricsRegistry
from pyformance.reporters.carbon_reporter import CarbonReporter

registry = MetricsRegistry()
__reporter__ = CarbonReporter(registry=registry,
                              reporting_interval=10,
                              prefix='sanic',
                              server='localhost',
                              port=2003)
__reporter__.start()
app = Sanic()

@app.route("/api/v1/foo", methods=["POST"])
def foo(request):

    timer = registry.timer(".foo")
    with timer.time():
        time.sleep(random.randint(1, 2))
        return json({"status": True})

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=20100, workers=1, access_log=False, debug=False)

README examples don't work

I'm trying to run the examples in the README file, and they're not working. This is a gist illustrating the issues I'm having (after pip install pyformance in a fresh virtualenv):

https://gist.github.com/emmett9001/2b8fd7d09220deb7d16c

Gist line 9: the output is 0, it should be close to 0.1
line 12: import fails
line 20: digging through the code, I tried this import, which works
line 37: running this example causes a NoneType error.

Am I using an outdated version of pyformance? Is there some fork that is more up to date?

Percentiles in histogram

I might be missing something but I don't see how to get percentiles from the histogram. This was a very useful feature in the original codahale metrics library.

Timer.clear() broken

Timer.clear() call raises exception because it calls to Meter.clear() which is not implemented

PyPI tarball omits doc and tests directories

I'm not sure if this omission it intentional or not. I would like to use the PyPI tarball as the source for a Linux distribution package, but the omission of the doc and tests directories makes it sub-optimal for this purpose. Would you be willing to include them? If not, then I'll use the github tarball instead. Either way, thanks for considering my request.

Release NewRelic reporter

Hi,
We noticed that the NewRelic reporter is missing from release 0.3.2 (and in pip. Is it stable to be used in production? If so, is it possible to publish the latest version?

Thanks

@omergertel

Measuring in Celery task

For a project of mine, I want to measure inside Celery tasks. But it seems that measured data from the Celery workers does not get reported. (It actually does get printed if I trigger that from within the worker)

A minimal example:
celerypyformance.py

from pyformance import MetricsRegistry
from pyformance.reporters import ConsoleReporter

from celery import Celery

app = Celery('celerypyformance', backend='redis://localhost', broker='redis://localhost')

registry = MetricsRegistry()
reporter = ConsoleReporter(registry, reporting_interval=2)
reporter.start()

registry.counter('started').inc()


@app.task
def task(argument):
    registry.counter('task.%s' % argument).inc()
    print('task.%s' % argument)

test.py

from celerypyformance import task

if __name__ == '__main__':
    task.delay("delayed")

It keeps printing only the start action, never the counter from within the task:

== 2016-11-29 10:49:20 ===================================
started:
               count = 1

[2016-11-29 11:49:21,986: WARNING/Worker-3] task.delayed
== 2016-11-29 10:49:22 ===================================
started:
               count = 1

To run:
Install Celery using pip, install redis. Start the redis server.
Run celery -A celerypyformance worker and leave this running while executing the next command.
Execute python test.py to execute the celery task.

Is there something I've forgotten or is there a small fix that can resolve this issue?
Thanks in advance

License restrictions

I'd like to consider using this package but need to know under what license it is published.

Tested in Docker?

Hello,

General question here, but has the code been tested with docker containers?

OpenTSDB_Reporter not reporting when running under python 3

opentsdb_reporter.py uses urllib.Request to send metrics to OpenTSDB:

                request = urllib.Request(self.url, data=json.dumps(metrics),
                                         headers={'content-type': "application/json"})

However, urllib2 in python2 expects a string, while urllib in python3 expects an array of bytes.

This creates runtime errors when running under Python3.

Incompatible with Python 3.12

The pyformance package is incompatible with a freshly installed Python 3.12 virtual environment, unless setuptools is also installed.

Steps to reproduce:

$ python3.12 -m venv env
$ source env/bin/activate
$ pip install pyformance
$ pip list

Package Version
------- -------
pip     24.0

Python 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyformance import MetricsRegistry
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/user/env/lib/python3.12/site-packages/pyformance/__init__.py", line 1, in <module>
    __import__('pkg_resources').declare_namespace(__name__)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'

Does'nt report the timer values in the influx

Hi,

I am using the influx reporter when I run timer function there insertion in record with time but all the values are null, I can see mean value of the function using getmean() but there is no reflection of the timer values in influx DB. The code I am using is given below

reporter = InfluxReporter(registry, reporting_interval=1, database='mydatabase', server='myserver.local', username='myusername', password='mypassword', port='8186')
reporter.start()


time1 = registry.timer("func1_calls")

# def testTimer():
#     with timer("test").time():
#         print("I am from with statement")
#         print(timer("test").get_mean())
#         reporter.report_now()

@time_calls
def func1():
    print("Hey! I am from function 1")

while True:
        time.sleep(1)
        func1()

Compatibility check with python 3.11x

Hello,
As per the PyPi documentation the latest python pyformance can support is Python 3.4
Does it also supports python 3.11x?
If no, is there any plan to support it?

Add custom naming to decorator

It would be nice if we could specify an optional metric name through the decorators instead of getting <FUNCTION_NAME>_calls.

Something like

    >>> from pyformance import counter, count_calls
    >>> @count_calls(metric_name="test.count")
    ... def test():
    ...     pass
    ...
    >>> for i in range(10):
    ...     test()
    ...
    >>> print counter("test.count").get_count()
    10

PEP8 compliance

Thanks for this library!

I've gone over the code base and noticed diversions from the recommended PEP8 standard. A few stuff are more critical than others for readability and code cleanliness.

I thought of applying an automatic code formatter to this code such as black, what are your thoughts @omergertel ?

HTTP Error 400: Bad Request when reporting to tsdb

Don't quite understand the OpenTSDBReporter interface, tsdb has no access control
http://opentsdb.net/docs/build/html/api_http/index.html#authentication-permissions, but we have to add write_key here. When experimenting it with our tsdb server, always got "HTTP Error 400: Bad Request", however, directly posting to tsdb w/o authentication looks good.

No idea what's wrong here, can anyone help out here?
Below is the code sample

from pyformance import MetricsRegistry
from pyformance.reporters import OpenTSDBReporter
import time

registry = MetricsRegistry()
reporter = OpenTSDBReporter(registry=registry,
                            reporting_interval=10,
                            url="http://${tsdb_host}/api/put",
                            application_name="app",
                            write_key="key")
reporter.start()
counter = registry.counter("test")

while True:
    time.sleep(2)
    counter.inc()
    print counter.get_count()

'tests' folder is included in pyformance python package

Hi,
'tests/' folder and its contents are included in the python package and are also installed when pyformance package is installed via pip. This is a major issue as it breaks the local test import of various clients of pyformance as python tries to look for tests in site packages first and this breaks the code of clients i.e. you cannot do 'import tests.' in your code anymore as your tests folder would never be looked instead pyformance's tests folder would be looked first.
Please exlude tests/ folder from setup.py of pyformance to rectify this issue.

Solution 1)
You can replace your package line like this in setup.py:
packages=find_packages(exclude=['tests*'])

Solution 2)
remove init.py from tests/ folder so that it becomes a normal directory and not a python package to exclude from pip install.

Possible to add a tag on the commit that was released as 0.4.0?

Thanks for the release! I was confused because I normally check the releases page for updates and didn't see it there. It also makes it easier to view a list of changed commits from the github UI.

Looks like it was probably f2c8459

Reporting via http

Would be very useful if there's a built-in way to fetch the metrics through http. That's a feature in Yammer's library (http://metrics.codahale.com/manual/servlet/#manual-servlet)

Log rotation for CsvReportrer

Would be nice to have an option to enable log rotation to csv reporting without having to configure logrotate for it. Is this planned for implemention?