leecampbell/hdrhistogram.net

Port of the HdrHistorgram to .NET/C#

hdrhistogram.net's Issues

Use common tool for perf tests

Consider replacing the hand rolled perf tests with a wrapper from https://www.nuget.org/packages/BenchmarkDotNet/

.NET Lib to create charts

Currently I believe that the only way for a .NET program to render the captured histograms into a chart is via the web project http://hdrhistogram.github.io/HdrHistogram/plotFiles.html.

It would be nice if this was ported to a .NET process (using the Drawing or WPF libs) to generate this stuff on the fly.

Create extreme packages

Create extreme packages that are designed for extreme performance cases

• only have a single implementation of a histogram defined
• remove the base class
• only implement the interface
• sealed class with no inheritance
• x64 and x86 (32bit) releases.
• no support for synchronization.

e.g. HdrHistogramLongx64.nupkg, HdrHistogramIntx64.nupkg, HdrHistogramShortx86.nupkg

Test to see if they provide significant improvements in

• footprint (dll/nuget package)
• throughput

These would be targets for either .NET platforms requiring extreme throughput (MMOG, Trading, etc), or that require very small footprint (UWA, RasberryPi, etc)

Create CI build

Create an automated build that

• compiles the code for each platform
• Runs portability analyzer (is this needed if we do a build for each platform?)
• runs all the tests
• packages and deploys

Consider using AppVeyor?

Create EBNF definition of the log format

https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form

https://gitter.im/HdrHistogram/HdrHistogram?at=571e8a9d47b4c6480ffa300d

Create automated Documentation generation

depends on #23 and #39

Could use https://readthedocs.org/ to host the generated docs.

Abstract recording precision from consumer

Make it easy to measure with this as the default

• Create constants of TicksInAnHour
• Set default value for outputValueUnitScalingRatio to go from ticks to milliseconds (should appease the Mort dev) eg. LongHistogram.OutputPercentileDistribution(Console.Out, outputValueUnitScalingRatio: TicksInAMillisecond);
• Create the MeasureAcount ex method

void RecordLatencyInTicks(this HistogramBase histogram, Action action)
{
    var start = Stopwatch.GetTimestamp();
    action();
    var elapsedTicks = Stopwatch.GetTimestamp() - start;
    histogram.RecordValue(elapsedTicks);
}

• Potentially rename our Histograms, or provide the factories that are appropriate.

Ideally we just want to get a Histogram

var histogram = Histogram.Create()
    .AsLong()   //WTF to I get from this? --Can I prove the tradeoffs? is short or int faster/Smaller? I hope not, then I can delete them.
    .WithTickPrecision()
    .AsThreadSafe()
    .Create()

then call

histogram.RecordLatency(()=>{...});

then output the results

historgram.OutputPercentileDistribution(Console.Out, OutputUnit.Milliseconds)

This allows consumers to not have to know about how it was constructed or how it will be charted

Create log reader

Create documentation standards

Adoption of a library can be greatly improved with quality documentation.

Providing guidelines on how to create consistent documentation can reduce rework and improve the feeling of a quality repository.

example of a set of guidelines : http://dotnet.github.io/orleans/Documentation-Guidelines

Why another .NET HdrHistogram ?

https://github.com/HdrHistogram/HdrHistogram/tree/master/src/main/csharp

Example of how to Send or save Histograms

This seems like it will become a popular thing and people are already asking for it

https://gitter.im/HdrHistogram/HdrHistogram?at=563f8127c712fe074e4e7101

Docs should use ticks not nanos

Update documentation to explain that .NET/Windows only supports a minimum precision of 100 nano-seconds (a tick).

Publish pdbs to SymbolServer.org

As part of the publishing of a release package, not only produce the Nuget package and publish that to Nuget.org, but also publish the symbols to SymbolServer.org

Benchmark against official C# port

Create an IHistogram interface

Create an IHistogram interface that the implementation all implement.
Then consumers can more easily replace entire implementations for testing, or even to just turn off (rotate with a NullImplementation).

Lzct SSE instruction when available

We currently have at the heart of the hot path a manual method for find the leading zero count.
This is used to identify the correct bucket to assign a recorded value.
Frustratingly, this is supported as an intrinsic instruction on most modern CPU architectures.

The code is found in Bitwise.NumberOfLeadingZeros, and has been isolated with the intent that it can be a single place to refactor/optimize if the opportunity arises.

Follow this .NET core issue for progress/resolution https://github.com/dotnet/corefx/issues/2209

Rationalise/Document the GoogleCharts folder

Document it or bin it

Document the Encoding format(s)

Mark the Nuget Package as "Unofficial"

As this isn't the official HdrHistogram.NET Repo*, we shouldn't be taking the HdrHistogram nuget package title.
It appears to have been taken anyway by the existing thing.
https://www.nuget.org/packages/HdrHistogram.NET/

Also worth removing the build time dependency that has made it into the nuspec file declaration

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2011/10/nuspec.xsd">
  <metadata>
    <!--TODO: rename to HdrHistogram -->
    <id>HdrHistogram</id>
    <version>2.0.75-beta</version>
    <title>HdrHistogram</title>
    ...
    <tags>HdrHistogram Histogram Instrumentation</tags>
    <!--TODO: remove this build time dependency -->
    <dependencies>
      <dependency id="AssemblyVersion.MSBuild" version="0.1.0" />
    </dependencies>
  </metadata>
</package>

*Official one is here - https://github.com/HdrHistogram/HdrHistogram.net

Create coding standards

To enable the community to contrubute to the repository in a consitent and predictable manner, it would be good if there was a set of standards for the project to adopt.

• Which style of C# to use (e.g. StyleCop defaults vs Resharper defaults)
• Expectation of XML documentation for public api
• Test coverage and style expectations.
• Platform support that is expected (.NET 20-4.5, Mono, Dnxcore etc?)

These expectation should be documented in the wiki

Split out Perf Tests from Unit tests

Rationalise/Document what the GnuPlot folder is for

Either document it or bin it

Add strong naming key to repo

It appears to be recommended practice to do so

• https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/strong-name-signing.md
• https://msdn.microsoft.com/en-us/library/wd40t7ad.aspx

Might also be relevant to investigate Open Source Signing when the lib adopts .NET Core

• https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/oss-signing.md

Update/Verify nuget package that can be used on PCL, .NET4, .NET 4.5 and .NET 5.0

I don't believe that the current nuget package can be used on PCL projects( e.g. ReactiveTrader). I believe that it has no dependency on platform specific features, so I think it would be safe to make it PCL compliant.

Correct the Nuget Version number in build artefacts

Auto-sizing

Guidance from Gil:

Auto-sizing is another useful thing… Not having to specify an initial range is useful for lazy folks (who are ok with resizing latencies in the recording path). It is also useful as a way to avoid overflowing wrongly-initial-sized histograms: unexpected large values result in a resize rather than an AIOOB exception. If your are ok with taking the latency hit (and potential mem size hit) for that, it's cleaner to code to.

Investigate SkinnyHistogram

I am not sure what the Skinny Histogram is and what it could offer the .NET project.

• [https://github.com/HdrHistogram/HdrHistogram/blob/master/HdrHistogram-benchmarks/src/main/java/org/HdrHistogram/SkinnyHistogram.java]
• [https://github.com/Searchlight/khronus/blob/master/khronus-core/src/main/scala/com/searchlight/khronus/model/SkinnyHistogram.scala]
• khronus/khronus#21 (comment)

Document/Visualize the internal bucket model

As asked and answered here [https://gitter.im/HdrHistogram/HdrHistogram?at=56db5610ddfe3d431627fa97]

Create a Recorder

Consider supporting Recorder (which supports multiple concurrent writers), but that will also require a ConcurrentHistogram.

Question: What is the Recorder?

I think he means Recorder.java and the specialist implementations such as SingleWriterRecorder.java and DoubleRecorder.java

You can see a good pattern for using a Recorder (along with a log writer) in jHiccup: https://github.com/giltene/jHiccup/blob/master/src/main/java/org/jhiccup/HiccupMeter.java#L626

SingleWriterRecorder is probably the most common use case, and the fastest in code. Recorder supports multiple writer threads, which is less commonly needed. DoubleRecorder will only be needed once you port DoubleHistogram.

Taken from HdrHistogram/HdrHistogram.NET#11

Create synchronized version for each storage type

Currently only the LongHistogram has a matching synchronized version. It seems reasonable to provide implementations for short and int as well.

This feature also seems to be an appropriate time to introduce a factory to help guide consumers as to which one is appropriate e.g.

var defaultHistogram = Histogram.Factory.Create(); //Creates a LongHistogram
var synchronizedLongHistogram = Histogram.Factory.AsThreadSafe().Create(); //Creates a SynchronizedLongHistogram
var longHistogram = Histogram.Factory.AsLongBacked().Create(); //Creates a LongHistogram
var synchronizedLongHistogram2 = Histogram.Factory.AsLongBacked().AsThreadSafe().Create(); //Creates a SynchronizedLongHistogram

var intHistogram = Histogram.Factory.AsIntBacked().Create(); //Creates a IntHistogram
var synchronizedIntHistogram = Histogram.Factory.AsIntBacked().AsThreadSafe().Create(); //Creates a SynchronizedIntHistogram

var shortHistogram = Histogram.Factory.AsShortBacked().Create(); //Creates a ShortHistogram
var synchronizedShortHistogram = Histogram.Factory.AsShortBacked().AsThreadSafe().Create(); //Creates a SynchronizedShortHistogram

Correct doc on Log Format

See @giltene note from gitter

https://gitter.im/HdrHistogram/HdrHistogram?at=5721811439700a780927e117

Create a Log writer

Support a log writer that would output V2 encoded histograms to a log file compatible with HistogramLogWriter

Instrument ASP.NET end point package

I assume that there is a simple way to add a middleware/handler/router/filter/thing to ASP.NET to allow HdrHistogram to record the time taken for the request to be processed.

If there is then this would be good to provide as a separate nuget package that web devs can just add and then wire up in a one-liner
It should

• record the service time,
• autorotate instances of HdrHistogram when writing to disk (to target directory)
• default to using the endpoint/method/action name as the key for grouping. (hsitograms can be merged at later date for higher level aggregates)

Create an example of how to rotate histograms to read or write but not both.

Add support for decoding V0 and V1 encodings

Test logs with V2, V1, and V0 encoded histograms are included in the Java repo under test/resources.

• V0 encoding
• V1 encoding
• V2 encoding

Create Nuget Package

If I delete the Reset features, can I squeeze out more performance?