omergertel / pyformance Goto Github PK
View Code? Open in Web Editor NEWPerformance metrics, based on Coda Hale's Yammer metrics
License: Other
Performance metrics, based on Coda Hale's Yammer metrics
License: Other
Would be beneficial for debugging if the module did some logging. For example, when Reporter starts for the first time or flushes to sink later on. Just the normal, basic stuff.
Hi,
Would you consider making a new release soon?
Many thanks,
David
I want to use the metrics and MetricsRegistry from this library and push the metrics to openTSDB. However, the metric only has a key which is not able to support tags from openTSDB. In that case, how can we create an OpenTSDBReporter which is similar to HostedGraphiteReporter for graphite? Thanks.
Hi,
Recently there were some fixes for the InfluxDB reporter added to Pyformance. However, these changes don't seem to be present in the version of Pyformance available via pip (0.3.4). I was hoping you could do a new release so we wouldn't have to install the latest code from Github.
Thanks!
Hello,
I think pyformance is great and I use it for some projects.
Therefore I fixed some stuff and added missing features.
The fork is available at:
https://github.com/hajs/pyformance/
The modifications are documented in the file CHANGES.
I would be glad if you consider merging my code.
There's currently no way to close the file handles opened by CsvReporter. This issue is demonstrated when running unit tests in a Windows environment:
python "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.3.2\helpers\pycharm\utrunner.py" C:\dev\git\pyformance\tests\test__csv_reporter.py true
Testing started at 11:50 PM ...
Error
Traceback (most recent call last):
File "C:\dev\git\pyformance\tests\test__csv_reporter.py", line 27, in tearDown
shutil.rmtree(self.path)
File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Luuseens\pyenvs\pyformance\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\-\\AppData\\Local\\Temp\\tmpd295pp0o\\gauge1.csv'
Process finished with exit code 0
Hi, thanks for this library.
Does pyformance support point tags? For example, AFAIK a graphite metric is a name, a timestamp (possibly implied), and a value, e.g. "cpu.idle [now] 42". Some metrics monitoring solutions allow point tags, for example you could have:
cpu.idle 42 host=myFoo env=prod
cpu.idle 23 host=myBar env=dev
and on the solution you could do aggregations by environment, or host, etc.
If pyformance does not support point tags, would allow a contribution adding support for them? Along with a reporter that knows how to stream points to a TCP socket?
The class ExpWeightedMovingAvg has a parameter which allows you to specify the period of reporting, yet there are 3 hardcoded instances in Meter:
self.m1rate = ExpWeightedMovingAvg(period=1, clock=self.clock)
self.m5rate = ExpWeightedMovingAvg(period=5, clock=self.clock)
self.m15rate = ExpWeightedMovingAvg(period=15, clock=self.clock)
Which results in only getting those metrics in the dashboard.
Would be great if you could also specify your own intervals. Something like Meter(clock=self._clock, [1, 5, 15, 30, 60])
Would this be useful? Seems fairly trivial to implement, would be happy to create the PR.
Is it possible this package could be added to conda forge? Although it is already on pip it would be great if conda users could also easily install pyformance!
Hi,
I just wrote simple script to get me started on pyformance, seems like influxdb reporter is not pushing any data to influxdb, my script is as below, can someone let me know if something is wrong with it?
from pyformance.registry import MetricsRegistry
from pyformance.reporters.influx import InfluxReporter
import time
registry = MetricsRegistry()
counter=registry.counter("test")
reporter=InfluxReporter(registry=registry)
reporter.start()
while True:
time.sleep(2)
counter.inc()
print counter.get_count()
It seems like you forgot to lock the Counter on dec()
:
pyformance/pyformance/meters/counter.py
Line 20 in b71056e
I am trying to add metrics to sanic application.
When running 1 worker everything looks fine, but when configuring more workers no metrics being sent to carbon.
import random
import time
from sanic import Sanic
from sanic.response import json
from pyformance import MetricsRegistry
from pyformance.reporters.carbon_reporter import CarbonReporter
registry = MetricsRegistry()
__reporter__ = CarbonReporter(registry=registry,
reporting_interval=10,
prefix='sanic',
server='localhost',
port=2003)
__reporter__.start()
app = Sanic()
@app.route("/api/v1/foo", methods=["POST"])
def foo(request):
timer = registry.timer(".foo")
with timer.time():
time.sleep(random.randint(1, 2))
return json({"status": True})
if __name__ == "__main__":
app.run(host='0.0.0.0', port=20100, workers=1, access_log=False, debug=False)
I'm trying to run the examples in the README file, and they're not working. This is a gist illustrating the issues I'm having (after pip install pyformance
in a fresh virtualenv):
https://gist.github.com/emmett9001/2b8fd7d09220deb7d16c
Gist line 9: the output is 0, it should be close to 0.1
line 12: import
fails
line 20: digging through the code, I tried this import, which works
line 37: running this example causes a NoneType
error.
Am I using an outdated version of pyformance? Is there some fork that is more up to date?
I might be missing something but I don't see how to get percentiles from the histogram. This was a very useful feature in the original codahale metrics library.
Timer.clear() call raises exception because it calls to Meter.clear() which is not implemented
I'm not sure if this omission it intentional or not. I would like to use the PyPI tarball as the source for a Linux distribution package, but the omission of the doc and tests directories makes it sub-optimal for this purpose. Would you be willing to include them? If not, then I'll use the github tarball instead. Either way, thanks for considering my request.
Hi,
We noticed that the NewRelic reporter is missing from release 0.3.2 (and in pip. Is it stable to be used in production? If so, is it possible to publish the latest version?
Thanks
For a project of mine, I want to measure inside Celery tasks. But it seems that measured data from the Celery workers does not get reported. (It actually does get printed if I trigger that from within the worker)
A minimal example:
celerypyformance.py
from pyformance import MetricsRegistry
from pyformance.reporters import ConsoleReporter
from celery import Celery
app = Celery('celerypyformance', backend='redis://localhost', broker='redis://localhost')
registry = MetricsRegistry()
reporter = ConsoleReporter(registry, reporting_interval=2)
reporter.start()
registry.counter('started').inc()
@app.task
def task(argument):
registry.counter('task.%s' % argument).inc()
print('task.%s' % argument)
test.py
from celerypyformance import task
if __name__ == '__main__':
task.delay("delayed")
It keeps printing only the start action, never the counter from within the task:
== 2016-11-29 10:49:20 ===================================
started:
count = 1
[2016-11-29 11:49:21,986: WARNING/Worker-3] task.delayed
== 2016-11-29 10:49:22 ===================================
started:
count = 1
To run:
Install Celery using pip, install redis. Start the redis server.
Run celery -A celerypyformance worker
and leave this running while executing the next command.
Execute python test.py
to execute the celery task.
Is there something I've forgotten or is there a small fix that can resolve this issue?
Thanks in advance
I'd like to consider using this package but need to know under what license it is published.
Hello,
General question here, but has the code been tested with docker containers?
opentsdb_reporter.py uses urllib.Request to send metrics to OpenTSDB:
request = urllib.Request(self.url, data=json.dumps(metrics),
headers={'content-type': "application/json"})
However, urllib2 in python2 expects a string, while urllib in python3 expects an array of bytes.
This creates runtime errors when running under Python3.
The pyformance
package is incompatible with a freshly installed Python 3.12 virtual environment, unless setuptools
is also installed.
Steps to reproduce:
$ python3.12 -m venv env
$ source env/bin/activate
$ pip install pyformance
$ pip list
Package Version
------- -------
pip 24.0
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyformance import MetricsRegistry
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/user/env/lib/python3.12/site-packages/pyformance/__init__.py", line 1, in <module>
__import__('pkg_resources').declare_namespace(__name__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'
Hi,
I am using the influx reporter when I run timer function there insertion in record with time but all the values are null, I can see mean value of the function using getmean() but there is no reflection of the timer values in influx DB. The code I am using is given below
reporter = InfluxReporter(registry, reporting_interval=1, database='mydatabase', server='myserver.local', username='myusername', password='mypassword', port='8186')
reporter.start()
time1 = registry.timer("func1_calls")
# def testTimer():
# with timer("test").time():
# print("I am from with statement")
# print(timer("test").get_mean())
# reporter.report_now()
@time_calls
def func1():
print("Hey! I am from function 1")
while True:
time.sleep(1)
func1()
Hello,
As per the PyPi documentation the latest python pyformance can support is Python 3.4
Does it also supports python 3.11x?
If no, is there any plan to support it?
It would be nice if we could specify an optional metric name through the decorators instead of getting <FUNCTION_NAME>_calls
.
Something like
>>> from pyformance import counter, count_calls
>>> @count_calls(metric_name="test.count")
... def test():
... pass
...
>>> for i in range(10):
... test()
...
>>> print counter("test.count").get_count()
10
Thanks for this library!
I've gone over the code base and noticed diversions from the recommended PEP8 standard. A few stuff are more critical than others for readability and code cleanliness.
I thought of applying an automatic code formatter to this code such as black, what are your thoughts @omergertel ?
Don't quite understand the OpenTSDBReporter interface, tsdb has no access control
http://opentsdb.net/docs/build/html/api_http/index.html#authentication-permissions, but we have to add write_key here. When experimenting it with our tsdb server, always got "HTTP Error 400: Bad Request", however, directly posting to tsdb w/o authentication looks good.
No idea what's wrong here, can anyone help out here?
Below is the code sample
from pyformance import MetricsRegistry
from pyformance.reporters import OpenTSDBReporter
import time
registry = MetricsRegistry()
reporter = OpenTSDBReporter(registry=registry,
reporting_interval=10,
url="http://${tsdb_host}/api/put",
application_name="app",
write_key="key")
reporter.start()
counter = registry.counter("test")
while True:
time.sleep(2)
counter.inc()
print counter.get_count()
Hi,
'tests/' folder and its contents are included in the python package and are also installed when pyformance package is installed via pip. This is a major issue as it breaks the local test import of various clients of pyformance as python tries to look for tests in site packages first and this breaks the code of clients i.e. you cannot do 'import tests.' in your code anymore as your tests folder would never be looked instead pyformance's tests folder would be looked first.
Please exlude tests/ folder from setup.py of pyformance to rectify this issue.
Solution 1)
You can replace your package line like this in setup.py:
packages=find_packages(exclude=['tests*'])
Solution 2)
remove init.py from tests/ folder so that it becomes a normal directory and not a python package to exclude from pip install.
Thanks for the release! I was confused because I normally check the releases page for updates and didn't see it there. It also makes it easier to view a list of changed commits from the github UI.
Looks like it was probably f2c8459
Would be very useful if there's a built-in way to fetch the metrics through http. That's a feature in Yammer's library (http://metrics.codahale.com/manual/servlet/#manual-servlet)
Would be nice to have an option to enable log rotation to csv reporting without having to configure logrotate for it. Is this planned for implemention?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.