Giter Site home page Giter Site logo

orleansdashboard's Introduction

Orleans Dashboard

.NET Core Node CI Nuget

Discuss on Discord


A dashboard for Microsoft Orleans which provides some simple metrics and insights into what is happening inside your Orleans appliction.

The dashboard is simple to set up and is designed as a convenience for developers to use whilst building Orleans applications. It is not designed as a replacement for a production monitoring system.

Installation

Using the Package Manager Console:

dotnet add package OrleansDashboard

Then add with programmatic configuration:

new HostBuilder()
  .UseOrleans(o => o.UseDashboard(options => { })
  .Build();

The dashboard should be installed on every silo in the cluster.

Start the silo, and open this url in your browser: http://localhost:8080

Please note, the dashboard registers its services and grains using ConfigureApplicationParts which disables the automatic discovery of grains in Orleans. To enable automatic discovery of the grains of the original project, change the configuration to:

new HostBuilder()
  .UseOrleans(o => {
    o.ConfigureApplicationParts(parts => parts.AddFromApplicationBaseDirectory());
    o.UseDashboard(options => { });
  })
  .Build();

Self-Hosted Dashboard

In some cases you may wish to disable the dashboard's own web server, and host the dashboard on your own web application.

You can configure to include the dashboard as middleware:

using Microsoft.AspNetCore.Http.Extensions;
using Orleans;
using Orleans.Hosting;

var builder = WebApplication.CreateBuilder();

builder.Host.UseOrleans(siloBuilder =>
{
  siloBuilder.UseLocalhostClustering();
  siloBuilder.UseDashboard(x => x.HostSelf = true);
});

var app = builder.Build();

app.Map("/dashboard", x => x.UseOrleansDashboard());

CPU and Memory Metrics on Windows

The CPU and Memory metrics are only enabled on Windows when you add the Microsoft.Orleans.OrleansTelemetryConsumers.Counters package and have registered an implementation of IHostEnvironmentStatistics such as with builder.UsePerfCounterEnvironmentStatistics() (currently Windows only). You also have to wait some time before you see the data.

CPU and Memory Metrics on Linux

Since version 2.3, Orleans includes an implementation of IHostEnvironmentStatistics for Linux in Microsoft.Orleans.OrleansTelemetryConsumers.Linux. To enable CPU and Memory metrics, install the NuGet package and add the implementation to the silo using siloBuilder.UseLinuxEnvironmentStatistics().

CPU and Memory Metrics on AWS ECS

A community-maintained implementation of IHostEnvironmentStatistics for AWS ECS is available in Orleans.TelemetryConsumers.ECS. To enable CPU and Memory metrics, install the NuGet package and add the implementation to the silo using siloBuilder.UseEcsTaskHostEnvironmentStatistics().

Configuring the Dashboard

The dashboard supports the following properties for the configuration:

  • Username : Set a username for accessing the dashboard (basic auth).
  • Password : Set a password for accessing the dashboard (basic auth).
  • Host : Host name to bind the web server to (default is *).
  • Port : Set the port number for the dashboard to listen on (default is 8080).
  • HostSelf : Set the dashboard to host it's own http server (default is true).
  • CounterUpdateIntervalMs : The update interval in milliseconds between sampling counters (default is 1000).
new HostBuilder()
  .UseOrleans(o => {
    o.UseDashboard(options => {
      options.Username = "USERNAME";
      options.Password = "PASSWORD";
      options.Host = "*";
      options.Port = 8080;
      options.HostSelf = true;
      options.CounterUpdateIntervalMs = 1000;
    })
  })
  .Build();

Note that some users have noticed performance degredation when using the dashboard. In this case it is recommended that you try increasing the CounterUpdateIntervalMS to 10000 to see if that helps.

Using the Dashboard

Once your silos are running, you can connect to any of them using your web browser: http://silo-address:8080/

If you've started the dashboard on an alternative port, you'll need to specify that instead.

The dashboard will also relay trace information over http. You can view this in the dashboard, or from the terminal: curl http://silo-address:8080/Trace

Grain Details

For the state output we use some conventions:

  1. Looks for a parameter less method "GetState". Example: TestStateInMemoryGrain.cs
  2. Looks for methods thats returns types that are declared as IPersistentState property in the concret grain type. Example: TestStateGrain.cs

For Compound Key grains pass the id in format "{id},{additionalKey}", example: 123,testing

Building the UI

This is only required if you want to modify the user interface.

The user interface is react.js, using browserify to compose the javascript delivered to the browser. The HTML and JS files are embedded resources within the dashboard DLL.

To build the UI, you must have node.js and npm installed.

To build index.min.js, which contains the UI components and dependencies, install the dependencies and run the build script using npm:

$ cd App
$ npm install
$ npm run build

This will copy the bundled, minified javascript file into the correct place for it to be picked up as an embedded resource in the .NET OrleansDashboard project.

You will need to rebuild the OrleansDashboard project to see any changes.

Testing the Dashboard

The Tests/TestHosts/ directory contains a number of preconfigured test application.

Try the Tests/TestHosts/TestHost project as a starting point.

Dashboard API

The dashboard exposes an HTTP API you can consume yourself.

DashboardCounters

GET /DashboardCounters

Returns a summary of cluster metrics. Number of active hosts (and a history), number of activations (and a history), summary of the active grains and active hosts.

{
  "totalActiveHostCount": 3,
  "totalActiveHostCountHistory": [ ... ],
  "hosts": [ ... ],
  "simpleGrainStats": [ ... ],
  "totalActivationCount": 32,
  "totalActivationCountHistory": [ ... ]
}

Historical Stats

GET /HistoricalStats/{siloAddress}

Returns last 100 samples of a silo's stats.

[
  {
    "activationCount": 175,
    "recentlyUsedActivationCount": 173,
    "requestQueueLength": 0,
    "sendQueueLength": 0,
    "receiveQueueLength": 0,
    "cpuUsage": 88.216095,
    "availableMemory": 5097017340,
    "memoryUsage": 46837756,
    "totalPhysicalMemory": 17179869184,
    "isOverloaded": false,
    "clientCount": 1,
    "receivedMessages": 8115,
    "sentMessages": 8114,
    "dateTime": "2017-07-05T11:58:11.39491Z"
  },
  ...
]

Silo Properties

GET /SiloProperties/{address}

Returns properties captured for the given Silo. At the moment this is just the Orleans version.

{
  "OrleansVersion": "1.5.0.0"
}

Grain Stats

GET /GrainStats/{grainName}

Returns the grain method profiling counters collected over the last 100 seconds for each grain, aggregated across all silos

{
    "TestGrains.TestGrain.ExampleMethod2": {
    "2017-07-05T12:23:31": {
    "period": "2017-07-05T12:23:31.2230715Z",
    "siloAddress": null,
    "grain": "TestGrains.TestGrain",
    "method": "ExampleMethod2",
    "count": 2,
    "exceptionCount": 2,
    "elapsedTime": 52.1346,
    "grainAndMethod": "TestGrains.TestGrain.ExampleMethod2"
  },
  "2017-07-05T12:23:32": {
    "period": "2017-07-05T12:23:32.0823568Z",
    "siloAddress": null,
    "grain": "TestGrains.TestGrain",
    "method": "ExampleMethod2",
    "count": 5,
    "exceptionCount": 4,
    "elapsedTime": 127.04310000000001,
    "grainAndMethod": "TestGrains.TestGrain.ExampleMethod2"
  },
  ...
}

Cluster Stats

GET /ClusterStats

Returns the aggregated grain method profiling counters collected over the last 100 seconds for whole cluster.

You should only look at the values for period, count, exceptionCount and elapsedTime. The other fields are not used in this response.

{
  "2017-07-05T12:11:32": {
    "period": "2017-07-05T12:11:32.6507369Z",
    "siloAddress": null,
    "grain": null,
    "method": null,
    "count": 32,
    "exceptionCount": 4,
    "elapsedTime": 153.57039999999998,
    "grainAndMethod": "."
  },
  "2017-07-05T12:11:33": {
    "period": "2017-07-05T12:11:33.7203266Z",
    "siloAddress": null,
    "grain": null,
    "method": null,
    "count": 10,
    "exceptionCount": 2,
    "elapsedTime": 65.87930000000001,
    "grainAndMethod": "."
  },
  ...
}

Silo Stats

GET /SiloStats/{siloAddress}

Returns the aggregated grain method profiling counters collected over the last 100 seconds for that silo.

You should only look at the values for period, count, exceptionCount and elapsedTime. The other fields are not used in this response.

{
  "2017-07-05T12:11:32": {
    "period": "2017-07-05T12:11:32.6507369Z",
    "siloAddress": null,
    "grain": null,
    "method": null,
    "count": 32,
    "exceptionCount": 4,
    "elapsedTime": 153.57039999999998,
    "grainAndMethod": "."
  },
  "2017-07-05T12:11:33": {
    "period": "2017-07-05T12:11:33.7203266Z",
    "siloAddress": null,
    "grain": null,
    "method": null,
    "count": 10,
    "exceptionCount": 2,
    "elapsedTime": 65.87930000000001,
    "grainAndMethod": "."
  },
  ...
}

Silo Stats

GET /SiloCounters/{siloAddress}

Returns the current values for the Silo's counters.

[
  {
    "name": "App.Requests.Latency.Average.Millis",
    "value": "153.000",
    "delta": null
  },
  {
    "name": "App.Requests.TimedOut",
    "value": "0",
    "delta": "0"
  },
  ...
]

Top Grain Methods

GET /TopGrainMethods

Returns the top 5 grain methods in terms of requests/sec, error rate and latency

{
  "calls": [
    {
      "grain": "TestGrains.TestGrain",
      "method": "ExampleMethod2",
      "count": 1621,
      "exceptionCount": 783,
      "elapsedTime": 343.75,
      "numberOfSamples": 100
    },
    {
      "grain": "TestGrains.TestGrain",
      "method": "ExampleMethod1",
      "count": 1621,
      "exceptionCount": 0,
      "elapsedTime": 91026.73,
      "numberOfSamples": 100
    }
    ...
  ],
  "latency": [ ... ],
  "errors": [ ... ],
}

Reminders

GET /Reminders/{page}

Returns the total number of reminders, and a page of 25 reminders. If the page number is not supplied, it defaults to page 1.

{
  "count": 1500,
  "reminders": [
    {
      "grainReference": "GrainReference:*grn/D32F2751/0000007b",
      "name": "Frequent",
      "startAt": "2017-07-05T11:53:51.8648668Z",
      "period": "00:01:00",
      "primaryKey": "123"
    },
    ...
  ]
}

Trace

GET /Trace

Streams the trace log as plain text in a long running HTTP request.

orleansdashboard's People

Contributors

alrz avatar andrewp-gh avatar benjaminpetit avatar carloslazo avatar dependabot[bot] avatar dixonds avatar euronay avatar frank-krick avatar galvesribeiro avatar jkonecki avatar jokin avatar justmara avatar koenbeuk avatar leonardosimoura avatar maxcalvin avatar meladkamari avatar melnikovig avatar n-sidorov avatar nzkyle avatar ph1ll avatar recursosonline avatar reubenbond avatar richorama avatar sebastianstehle avatar sebferraro avatar seniorquico avatar tanordheim avatar wangjia184 avatar yevhen avatar zzzasdfghjklzzz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

orleansdashboard's Issues

defining the port is case sensitive

<BootstrapProviders>
  <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" port="8088" />
</BootstrapProviders>

port must be lowercase. Perhaps just update the readme.md

Support for older Orleans releases

Would it be possible to have a version that supports .NET 4.5 and Orleans 1.0.10.0 ?

I have had a go at downgrading it without success. The dashboard would be really useful in our environment.

All grains fail during OnActiveAsync

Hi,

I've recently updated our App to Orleans 1.3.0 and it works fine. However, I wanted to try your Dashboard (especially the latency and throughout metrics, which I love! :) ) but when i add Dasboard to the project, any call to any Grain fail.

Log:
[2016-10-20 13:00:46.826 GMT    28  INFO    100000  SiloControl 135.128.216.44:11111]   GetSimpleGrainStatistics    
[2016-10-20 13:00:46.822 GMT    21  ERROR   100513  Catalog 135.128.216.44:11111]   !!!!!!!!!! Error calling grain's OnActivateAsync() method - Grain type = XXXXXX.Statistics.QueueLengthHistoryMaintenanceGrain Activation = [Activation: S135.128.216.44:11111:214664102*grn/CCD89FBB/00000000@e699b3ac #GrainType=XXXXX.Statistics.QueueLengthHistoryMaintenanceGrain Placement=RandomPlacement State=Activating]   
Exc level 0: System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

Server stack trace: 
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Orleans.InterceptedMethodInvokerCache.GetInterfaceToImplementationMap(Int32 interfaceId, Type implementationType)
   at Orleans.InterceptedMethodInvokerCache.CreateInterceptedMethodInvoker(Type implementationType, Int32 interfaceId, IGrainMethodInvoker invoker)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Orleans.InterceptedMethodInvokerCache.GetOrCreate(Type implementationType, Int32 interfaceId, IGrainMethodInvoker invoker)
   at Orleans.Runtime.InsideRuntimeClient.InvokeWithInterceptors(IAddressable target, InvokeMethodRequest request, IGrainMethodInvoker invoker)
   at Orleans.Runtime.InsideRuntimeClient.<Invoke>d__57.MoveNext()

Exception rethrown at [0]: 
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at XXXXX.Statistics.QueueLengthHistoryMaintenanceGrain.<ReminderInterval>d__8.MoveNext() in C:\XXXXXX\Statistics\QueueLengthHistoryMaintenanceGrain.cs:line 36
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at XXXXXX.ReminderRepeaterBaseGrain.<OnActivateAsync>d__6.MoveNext() in C:\XXXXXX\ReminderRepeaterBaseGrain.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Catalog.<CallGrainActivate>d__66.MoveNext()
[2016-10-20 13:00:46.879 GMT    28  WARNING 100534  Catalog 135.128.216.44:11111]   Failed to InvokeActivate for [Activation: S135.128.216.44:11111:214664102*grn/CCD89FBB/00000000@e699b3ac #GrainType=XXXXXX.Statistics.QueueLengthHistoryMaintenanceGrain Placement=RandomPlacement State=Invalid].  
Exc level 0: System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

In our project we use Bootstrap to init IoC container (Unity), we plan to change it to the way how DI works in Orleans now but we didn't have time to do it so (we upgraded from Orleans 1.0.10 where DI was not in a way how it's now).

If I comment out this line from config:

 <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" />

it works fine so it seems that Dashboard Bootstrap somehow mess things up?

Auto-scroll for Log Stream

First of all, I love this new screen Log Stream, very useful!

There is a minor improvement though that would be useful IMO - auto-scroll the view to show the latest logs always. I guess, it could be either the default behavior or some checkbox at the top

OrleansDashboard.SiloGrain has called DeactivateOnIdle from within OnActivateAsync, which is not allowed.

[2016-06-01 00:36:53.369 GMT 11 ERROR 100513 Catalog 172.18.124.93:11111] !!!!!!!!!! Error calling grain's OnActivateAsync() method - Grain type = OrleansDashboard.SiloGrain Activation = [Activation: S172.18.124.93:11111:202434420*grn/ACD5BA64/00000000+172.18.124.93:11111@202429561@6623488c #GrainType=OrleansDashboard.SiloGrain Placement=RandomPlacement State=Activating]

Exc level 0: System.InvalidOperationException: Activation [Activation: S172.18.124.93:11111:202434420*grn/ACD5BA64/00000000+172.18.124.93:11111@202429561@6623488c #GrainType=OrleansDashboard.SiloGrain Placement=RandomPlacement State=Activating] has called DeactivateOnIdle from within OnActivateAsync, which is not allowed.

How are the 'tags' implemented?

Next to each grain type, the dashboard shows what look like tags, e.g. "System Grain" or "Dashboard Grain" (see screenshot).

How do these work - is it actually possible to tag grains?

Silo uptime and local time

The silo uptime time seems to wrong. See the screenshot:
uptime_time

The screenshot was taken during the deployment when the first silo just restarted. I believe it has to do with the local timezone of the PC where the screenshot was taken - it is UTC+3

It is still 1.3.1, maybe it is not an issue in newer versions.

support custom user ordering of grain method invocation charts

The order grain methods appear in the dashboard won't match the expectations and needs of developers and IT admins unless they can change it and have those changes tracked by Orleans Dashboard. Some grain methods occur in pairs or larger groups, and some calls or groups of calls are related to different stages of custom business processes. Being able to put those calls in an order that represents the understanding of the organization monitoring the data has value.

Let the user move grain method charts up or down on the page. Save that sort order so all new dashboard sessions assume the new sort order.

Add reminders view

It would be beneficial, I think, to have another tab for reminders where you could see the list of them. Each entry could show the grain type, the grain key, the reminder name, its period and the next fire time. And finally, there could be Delete button to remove the selected reminder - it would be super helpful for orphaned reminders after grain refactoring.

System.MissingMemberException: The server factory could not be located for the given input: Nowin

Hi,

I have implement Orleans Dashboard for quite sometime and everything is working fine (the last working version is 1.3.4) but recently I upgraded my project to VS2017 and I face the following error
System.MissingMemberException: The server factory could not be located for the given input: Nowin

cant seems to find any solution online, please advice.

I am using the programmatic configuration:

var siloHost = new SiloHost(...);
siloHost.InitializeOrleansSilo();
siloHost.Config.Globals.RegisterDashboard(); // port, username and password can also be supplied
siloHost.StartOrleansSilo();

Error

[16:56:49 INF] Runtime - 127.0.0.1:11111: Loaded provider of type OrleansDashboard.Dashboard Name=Dashboard
[2016-12-20 08:56:49.862 GMT    22      INFO    103102  ProviderLoader/IBootstrapProvider       127.0.0.1:11111]
Loaded provider of type OrleansDashboard.Dashboard Name=Dashboard
[16:56:49 INF] Runtime - 127.0.0.1:11111: Calling Init for IBootstrapProvider classes
[2016-12-20 08:56:49.873 GMT    22      INFO    100437  BootstrapProviderManager        127.0.0.1:11111]        Calling Init for IBootstrapProvider classes
[16:56:49 ERR] Provider - 127.0.0.1:11111: System.MissingMemberException: The server factory could not be located for the given input: Nowin
   at Microsoft.Owin.Hosting.Engine.HostingEngine.ResolveServerFactory(StartContext context)
   at Microsoft.Owin.Hosting.Engine.HostingEngine.Start(StartContext context)
   at OrleansDashboard.Dashboard.<Init>d__8.MoveNext()
OrleansHost.exe Error: 0 : [2016-12-20 08:56:49.939 GMT    22   ERROR   10001   Dashboard       127.0.0.1:11111]
!!!!!!!!!! System.MissingMemberException: The server factory could not be located for the given input: Nowin
   at Microsoft.Owin.Hosting.Engine.HostingEngine.ResolveServerFactory(StartContext context)
   at Microsoft.Owin.Hosting.Engine.HostingEngine.Start(StartContext context)
   at OrleansDashboard.Dashboard.<Init>d__8.MoveNext()
[2016-12-20 08:56:49.939 GMT    22      ERROR   10001   Dashboard       127.0.0.1:11111]        !!!!!!!!!! System.MissingMemberException: The server factory could not be located for the given input: Nowin
   at Microsoft.Owin.Hosting.Engine.HostingEngine.ResolveServerFactory(StartContext context)
   at Microsoft.Owin.Hosting.Engine.HostingEngine.Start(StartContext context)
   at OrleansDashboard.Dashboard.<Init>d__8.MoveNext()
[16:56:50 INF] Runtime - 127.0.0.1:11111: SystemStatus=Running
[16:56:50 INF] Runtime - 127.0.0.1:11111: Starting AsyncAgent Runtime.Messaging.GatewayAcceptor on managed thread 28
OrleansHost.exe Information: 0 : [2016-12-20 08:56:50.042 GMT     1     INFO    100294  SystemStatus    127.0.0.1:11111]        SystemStatus=Running

Support for Silo Cluster?

Hi,

Thanks for this very handy tool.
I have tried this with a standalone silo, where this looks perfectly fine; but I would like to know if this code also works with Cluster of Silos. Thanks.

Regards,

Pravin

Better support for generic grains

i.e.

public interface IGenericGrain<GenericType> : IGrainWithIntegerKey
{
    Task DoSomething();
}

shows up like

MyNamespace.MyGrain`1[[MyNamespace.MyGenericType, MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]

on the grain type page, support higher-density display of (shorter) grain method invocation charts

With a dozen or two grain methods being called, the beautiful tall charts when combined can create a very long web page which lets the user view only a few grain method call patterns at a time. In Chrome I can zoom in to make things larger, but apparently not smaller.

It'd be great to either display shorter charts, to offer a selection of chart sizes on the page, or to display an initial list of shorter charts on the page but let the user hover or select one of them to enlarge it for a clearer view.

Silo will not properly shutdown if viewing logstream

BACKGROUND

Using Orleans.Dashboard 1.4.7
Using Orleans 1.4.1

PROBLEM

The silo will not stop when viewing the log stream page

IMPACT

In an automated CI/CD pipeline environment, releases can fail if someone happens to be viewing the log stream. This is unacceptable.

STEPS TO REPRODUCE

  1. start silo
  2. navigate to the dashboard, any page except 'Log Stream'
  3. stop silo
    ==> observe the silo cleanly stops (as per log 1 below)
  4. re-start silo
  5. navigate to the dashboard, to 'Log Stream' page
  6. stop silo
    ==> observe the silo does not stop (as per log 2 below)

CONTEXT
Shutting down the silo is done by the following code.
Incidentally, the exception does not bubble up from ShutdownOrleansSilo().
In fact, the call to ShutdownOrleansSilo() never even returns

   public void Stop()
   {
       this.logger.Info($"Stopping Orleans Silo...");
       try
       {
           if (this.siloHost != null)
           {
               this.siloHost.ShutdownOrleansSilo();
               this.siloHost.Dispose();
               this.siloHost = null;
               this.logger.Info($"Orleans silo '{this.siloHost.Name}' shutdown.");
           }

           this.logger.Info("Orleans Silo Stopped");
       }
       catch (Exception exc)
       {
           this.logger.Fatal("Orleans Host Stop {@Error}: ", exc);

           if (this.siloHost != null)
           {
               this.siloHost.Dispose();
               this.siloHost = null;
           }
       }
   }

LOGS


log 1 - when NOT viewing the 'Log Stream' page --

[2017-04-20 22:52:23.040 GMT 7 INFO 100000 ManagementGrain 10.88.42.55:22222] GetDetailedHosts onlyActive=True
[2017-04-20 22:52:23.065 GMT 10 INFO 100000 SiloControl 10.88.42.55:22222] GetSimpleGrainStatistics
[2017-04-20 22:52:23.105 GMT 5 INFO 100294 SystemStatus 10.88.42.55:22222] SystemStatus=Stopping
[2017-04-20 22:52:23.107 GMT 5 INFO 100414 Silo 10.88.42.55:22222] Silo starting to Stop()
[2017-04-20 22:52:23.110 GMT 6 INFO 100607 MembershipOracle 10.88.42.55:22222] -Stop
[2017-04-20 22:52:23.122 GMT 9 INFO 102927 Orleans.Runtime.ReminderService.LocalReminderService 10.88.42.55:22222] Stopping Orleans.Runtime.ReminderService.LocalReminderService grain service
[2017-04-20 22:52:23.151 GMT 5 INFO 100413 Silo 10.88.42.55:22222] Silo is Stopped()
[2017-04-20 22:52:23.163 GMT 10 INFO 100609 MembershipOracle 10.88.42.55:22222] -KillMyself
[2017-04-20 22:52:23.174 GMT 6 INFO 100328 Runtime.Scheduler.WorkerPoolThread/0 10.88.42.55:22222] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/0 that runs on managed thread 6
[2017-04-20 22:52:23.174 GMT 8 INFO 100328 Runtime.Scheduler.WorkerPoolThread/2 10.88.42.55:22222] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/2 that runs on managed thread 8
[2017-04-20 22:52:23.174 GMT 7 INFO 100328 Runtime.Scheduler.WorkerPoolThread/1 10.88.42.55:22222] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/1 that runs on managed thread 7
[2017-04-20 22:52:23.174 GMT 10 INFO 100328 Runtime.Scheduler.WorkerPoolThread/System.5 10.88.42.55:22222] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/System.5 that runs on managed thread 10
[2017-04-20 22:52:23.176 GMT 9 WARNING 101222 Scheduler.ClientObserverRegistrar.WorkItemGroup 10.88.42.55:22222] Thread <Runtime.Scheduler.WorkerPoolThread/3, ManagedThreadId=9, Executing WorkItem=SystemWorkItemGroup:Name=ClientObserverRegistrar,WorkGroupStatus=Running Executing for 00:00:00.0050081. .> is exiting work loop due to cancellation token. WorkItemGroup: SystemWorkItemGroup:Name=ClientObserverRegistrar,WorkGroupStatus=Running, Have 0 work items in the queue.
[2017-04-20 22:52:23.178 GMT 9 INFO 100328 Runtime.Scheduler.WorkerPoolThread/3 10.88.42.55:22222] Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/3 that runs on managed thread 9
...
... lots more lines
...
SystemTarget..Current=3
Watchdog.NumHealthChecks.Current=0

[2017-04-20 22:52:23.284 GMT 5 INFO 100294 SystemStatus 10.88.42.55:22222] SystemStatus=Terminated


log 2 - when viewing the 'Log Stream' page --

[2017-04-21 04:14:05.855 GMT 10 INFO 100000 SiloControl 10.88.42.55:22222] GetSimpleGrainStatistics
[2017-04-21 04:14:06.178 GMT 58 INFO 100294 SystemStatus 10.88.42.55:22222] SystemStatus=ShuttingDown
[2017-04-21 04:14:06.180 GMT 58 INFO 100417 Silo 10.88.42.55:22222] Silo starting to Shutdown()
[2017-04-21 04:14:06.182 GMT 6 INFO 100606 MembershipOracle 10.88.42.55:22222] -ShutDown
[2017-04-21 04:14:06.205 GMT 7 INFO 102927 Orleans.Runtime.ReminderService.LocalReminderService 10.88.42.55:22222] Stopping Orleans.Runtime.ReminderService.LocalReminderService grain service
[2017-04-21 04:14:06.211 GMT 58 INFO 100545 Catalog 10.88.42.55:22222] DeactivateAllActivations.
[2017-04-21 04:14:06.221 GMT 58 INFO 100541 Catalog 10.88.42.55:22222] DeactivateActivations: total 3 to shutdown, out of them 3 promptly, 0 later when become idle and 0 are already being destroyed or invalid.
[2017-04-21 04:14:06.227 GMT 58 INFO 100503 Catalog 10.88.42.55:22222] Starting DestroyActivations #0 of 3 activations
[2017-04-21 04:14:06.259 GMT 83 WARNING 100502 Catalog 10.88.42.55:22222] UnregisterManyAsync 2 failed.
Exc level 0: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<UnregisterManyAsync>d__108.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<<QueueTask>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Catalog.<FinishDestroyActivations>d__75.MoveNext() [2017-04-21 04:14:06.258 GMT 5 WARNING 100502 Catalog 10.88.42.55:22222] UnregisterManyAsync 1 failed. Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__108.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Catalog.d__75.MoveNext()
[2017-04-21 04:14:06.274 GMT 83 INFO 101033 Dispatcher 10.88.42.55:22222] Forwarding 1 requests to old address S10.88.42.55:22222:230444020*grn/FFE8F7B3/00000000@99c29dbc after Finished Destroy Activation.
... NO FURTHER LOG LINES

Dashboard no longer exposes itself on public hostname

After updating from version 1.4x to 1.5 the Dashboard no longer exposes itself on the public IP / Hostname.

It would be good if it either
[a] exposed itself automatically or
[b] allowed programmatic options to set hostname / ip

"unknown comma" on generic Grain

I have a grain that's defined like HelloGrain : Grain, IHelloGrain<T, U> and I receive a console log message: "unknown comma" when this type of Grain is parsed by the TypeFormatter. For your reference, this type of generic functionality has been possible in Orleans since 1.4.0.

Dashboard startup failing after Orleans 1.5 update

The log message below is produced after the latest orleans update to 1.5. However copying libuv.dll into the bin directory seems the solve to startup problem as a work around.

xxx.Orleans.Silo.exe Error: 0 : [2017-07-10 23:51:17.297 GMT 27 ERROR 10001 Dashboard 127.0.0.1:11111] !!!!!!!!!! System.AggregateException: One or more errors occurred. ---> System.DllNotFoundException: Unable to load DLL 'libuv': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.NativeMethods.uv_loop_size() at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvLoopHandle.Init(Libuv uv) at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object parameter) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Start(Int32 count) at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Start[TContext](IHttpApplication1 application)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.Start()
at OrleansDashboard.Dashboard.d__15.MoveNext()
---> (Inner Exception #0) System.DllNotFoundException: Unable to load DLL 'libuv': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.NativeMethods.uv_loop_size()
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvLoopHandle.Init(Libuv uv)
at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object parameter)<---`

Orleans Dashboard on Service Fabric Host

Hi,
I got this when I tested the Dashborad With Orleans on Service Fabric.

How can I solve this issue?

Message": "[[email protected]:30068@PID:7208] Caught and ignored exception: System.InvalidOperationException with mesagge: The current membership oracle does not support detailed silo status reporting. thrown from timer callback GrainTimer. TimerCallbackHandler:OrleansDashboard.DashboardGrain->System.Threading.Tasks.Task Callback(System.Object)
Exception: System.InvalidOperationException: The current membership oracle does not support detailed silo status reporting.

All the help is appreciated

Thanks

Unhandled exception in GrainProfiler causes silo fail

This is the last log before the silo crashed:

[2016-11-07 11:01:06.930 GMT    72	ERROR  	100324	Silo	XXX.XXX.XXX.XXX:11111]	!!!!!!!!!! Called DomainUnobservedExceptionHandler with context Name:RdRuntime
There are no context policies.
.	
Exc level 0: System.AggregateException: One or more errors occurred.
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at OrleansDashboard.GrainProfiler.ProcessStats(Object state)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
Exc level 1: System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request SXXX.XXX.XXX.XXX:11111:216212275ProviderManagerSystemTarget@S00000013->SXXX.XXX.XXX.XXX:11111:216212275*grn/FFE8F7B3/00000000@c4d2e8a5 #18592: global::OrleansDashboard.IDashboardGrain:SubmitTracing(). Target History is: <SXXX.XXX.XXX.XXX:11111:216212275:*grn/FFE8F7B3/00000000:@c4d2e8a5>.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at OrleansDashboard.GrainProfiler.<>c__DisplayClass14_0.<<ProcessStats>b__0>d.MoveNext()

It seems that unhandled exception in GrainProfiler.ProcessStats called on timer caused the whole thing to crash

add virtual streams to the dashboard

Virtual streams would be a good thing to monitor. Which grains or grain methods are publishing messages to which virtual streams? How frequently are messages being published to each of the virtual streams?

I imagine a list of virtual streams like grain methods all on a page together, with message publish events visualized in the time-series chart, with metrics for number of messages published per second, etc.

If there are large numbers of virtual streams, some way to aggregate them, organize them in a tree/outline, etc., would likely be needed. I don't have all the answers, I just think it would be great to see activity on our virtual streams.

Any ideas?

show grain method invocation activity per grain type (across all methods) on dashboard home page

The Orleans home page has two sections: "Grains" with a chart showing number of grain activations over time, and "Silos" with a chart of the number of silos over time.

For each grain type listed, display a short chart of the number of grain method invocations over time for each grain type: the total number of grain method invocations, regardless of which grain method was called.

Add horizontal time axis to dashboard charts

It'd be nice to see the time indicated along the bottom of the chart, showing how long various scale-up operations took. Maybe hover over a point on the graph to see the UTC DateTime displayed or something.

Dashboard Console.Writeline

Is there a way to prevent the console output? We have removed all of the Orleans log messages from the console since we added a custom TraceListener to use NLog. Once those logs were removed, I noticed that we get a console output every few seconds. I assume this is caused by the dashboard?

2017-01-25 13:52:07.3060|INFO|Demon.Mission.Host.vshost.exe|[2017-01-25 21:52:07.304 GMT 21 INFO 100000 ManagementGrain 127.0.0.1:11111] GetDetailedHosts onlyActive=True

2017-01-25 13:52:07.3320|INFO|Demon.Mission.Host.vshost.exe|[2017-01-25 21:52:07.310 GMT 22 INFO 100000 SiloControl 127.0.0.1:11111] GetSimpleGrainStatistics

Improvements to grain tables

Suggested by @attilah :

I'd separate the system and user grains (by a builtin list?) and truncate the namespaces and show the full type name as tooltip. Also order them by their name (later make it sortable, filterable). Also later they can be click thru links to get the state or other info from the Grain itself.

Track OnActivateAsync on the grain view

Currently methods stats shown on the grain view are not including OnActivateAsync which itself could take some time. Is there an easy way to add it and probably OnDeactivateAsync?

Unhandled exceptions in DashboardController

Getting this from logs:

[2016-12-05 20:06:07.985 GMT     2	ERROR  	100102	Silo	XXX.XXX.XXX.XXX:11111]	
!!!!!!!!!! Silo caught an UnobservedException with context==null.	
Exc level 0: System.TimeoutException: Response did not arrive on time in 00:01:00 for message: Request SXXX.XXX.XXX.XXX:11111:218664204ProviderManagerSystemTarget@S00000013->SXXX.XXX.XXX.XXX:11111:218664204*grn/FFE8F7B3/00000000@d1409a8e #4038: global::OrleansDashboard.IDashboardGrain:GetCounters(). Target History is: <SXXX.XXX.XXX.XXX:11111:218664204:*grn/FFE8F7B3/00000000:@d1409a8e>.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at OrleansDashboard.DashboardController.<>c__DisplayClass11_0.<<GetDashboardCounters>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at OrleansDashboard.DashboardController.<GetDashboardCounters>d__11.MoveNext()

Wouldn't it be better to catch and log exceptions in DashboardController without passing them through as unobserved exceptions?

Cannot load IDashboardGrain when deployed to Azure

I get the following error:

[2016-10-18 13:24:37.357 GMT    52  ERROR   103109  ProviderLoader/IBootstrapProvider   10.220.7.4:11111]   !!!!!!!!!! Exception initializing provider Name=Dashboard Type=[Dashboard, OrleansDashboard.Dashboard]  
Exc level 0: System.InvalidOperationException: Cannot find generated GrainReference class for interface 'OrleansDashboard.IDashboardGrain'
   at Orleans.GrainFactory.MakeCaster(Type interfaceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Orleans.GrainFactory.Cast(IAddressable grain, Type interfaceType)
   at Orleans.GrainFactory.Cast[TGrainInterface](IAddressable grain)
   at OrleansDashboard.Dashboard.Init(String name, IProviderRuntime providerRuntime, IProviderConfiguration config)
   at Orleans.Providers.ProviderLoader`1.<InitProviders>d__7.MoveNext()

Any ideas what could be wrong or how can I track the source of the issue?

Dashboard throws an exception when hosted in Service Fabric

[2017-05-15 14:46:52.796 GMT    16    ERROR      101413    GrainTimer    127.0.0.1:30012]    !!!!!!!!!! Caught and ignored exception: System.NotImplementedException with mesagge: No membership table provider found for LivenessType=NotSpecified thrown from timer callback GrainTimer. TimerCallbackHandler:OrleansDashboard.DashboardGrain->System.Threading.Tasks.Task Callback(System.Object)    
Exc level 0: System.NotImplementedException: No membership table provider found for LivenessType=NotSpecified

Server stack trace: 
   at Orleans.Runtime.MembershipService.MembershipTableFactory.<GetMembershipTable>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Management.ManagementGrain.<GetHosts>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Management.ManagementGrain.<GetTotalActivationCount>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at OrleansDashboard.GrainProfiler.<InvokeInterceptor>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.InsideRuntimeClient.<Invoke>d__61.MoveNext()

Exception rethrown at [0]: 
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at OrleansDashboard.DashboardGrain.<Callback>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.GrainTimer.<ForwardToAsyncCallback>d__21.MoveNext()

Upgrade to new invoker API

...from build:

GrainProfiler.cs(30,37): warning CS0618: 'IProviderRuntime.GetInvokeInterceptor()' is obsolete: 'Retrieve InvokeInterceptor instances from the ServiceProvider property.' [C:\projects\orleansdashboard\OrleansDashboard\OrleansDashboard.csproj]
GrainProfiler.cs(31,13): warning CS0618: 'IProviderRuntime.SetInvokeInterceptor(InvokeInterceptor)' is obsolete: 'Register InvokeInterceptor instances with the service provider during configuration.' 

HTTP Errors

Return exception information in body when there's an HTTP error

Invalid total request/second values

It looks like the total requests/second value is the average from individual, per-silo requests/second values. Correct me please if I am wrong, but if one silo gives X requests/second and the other one - Y requests/second, doesn't it mean the totally two silos serve X+Y requests per second?

rps

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.