electnewt/distribt

Hey 👋, I am Ivan Abad

Senior Software Engineer mainly working in .NET building distributed systems.

I am creating software engineering videos on my YouTube channel. I do specialize in .NET specially C# and Blazor.

All the code for my videos can be found in my github repos, but you can find the full explanation of the implementation on my Website.

Side projects 👯

Apart from everyting I write for my website, I mainly maintain for the open source world:

• EjemploRop: Implementation of Raiway oriented programming in C#.

Alternatively I have a few more Projects that I've been working on.

• SocialGiveaway: App which allows to do raffles using different social network APIs.
• Distribt: Repo which implements a distributed system.
• Payments With Stripe: Complete course that shows how to integrate a Stripe API on a C# application.

And a book 📖

• Guía completa de desarrollo full stack con .NET: I wrote a book about building applications with .NET.

You are free to join in the development to improve them.

Contact ⚡

You can contacte me on LinkedIn, Twitter, or using my Website.

distribt's Issues

SecretManager: Separate engines

at the moment all the "get" are under ISecretManager but vault acts that you can have mutiple engines.

.get<T> should be to get the default key/value;
But to access rabbitMQ credentials should be something like
.RabbitMQ.Get rather than have all the methods directly in ISecretManager as if it grows with 50 services it will be unmanageable

abstract all HttpCalls to Other microservices to a shared project

when a call to a known microservice is done it should be abstracted, i can have an interface or similar whith all the endpoints, or create the client automatically based in the openapi spec or similar.

add unit testing in ProductNameService

ProductNameService has quite a lot of logic, it will be great to have at least unit testing.

Location:

Distribt.Services.Orders.BusinessLogic.Services.External;

Prepare the enviornment to build and run integration tests

Atm the code only "supports" unit tests, it should support also integration tests (building the containers, etc in docker)

Communication: Improve how the handlers get added; atm

The current way of adding the handlers is crap, because we are instanciating them, which limit us way to much for dependency injection and complexity.

This has to be fixed, and soon.

Add new MessageModels easily

The code to select the queue is hard coupled with the type of message.
We can use a list of Scopes (aka Model name) where we can define the consumer and subscriber queue names

-return (typeof(TMessage) == typeof(IntegrationMessage)
-           ? _settings.Consumer?.IntegrationQueue
-           : _settings.Consumer?.DomainQueue)
-       ?? throw new ArgumentException("please configure the queues on the appsettings");
+var scope = typeof(IntegrationMessage).Name.ToString();
+var queueSettings = _settings.Queues.FirstOrDefault(q => q.Scope == scope);
+return queueSettings?.Consumer
+    ?? throw new ArgumentException("please configure the queues on the appsettings");

For this, we need to change the AppSettings.json file

"RabbitMQ": {
-  "Publisher": {
-    "IntegrationExchange": "api.public.exchange"
-  }
+ "Queues": [
+   {
+     "Scope": "IntegrationMessage",
+     "Publisher": "api.public.exchange"
+   }
+ ]
}

What do you think?

include host and port of graylog inside consul

they are in the appsettings right now

Add tests for Distribt.shared.communication.RabbitMQ

rabbitMQ library should contain tests, mainly integration, but also unit.

investigate the use of inject IMongoDatabase on the event store

So at the moment I am generating IMongoDatabase in the MongoDbDependencyInjection whic is good, then in the eventSourcing i am using IOptions<databaseinfo> rather than inject IMongoDatabase;

thi is because, if I inject the interface rather than the configuration I will not be able to add a mongo database and an event store in the same project.

Investigate both points and leave a clear note about it (documented)

Documentation: document distribt.shared.secret

RabbitMQ: binding the queues on a lazy way

if a queue/exange does not exist the process will fail.

one solution is "lazy" creating those queues/exanges. but the proper way should be infrastrcuture wise.
But at the same time we are binding the queues and the exanges already in the code, so, why not to creating them too.

kafka health check not working

seems like the kafka healthcheck is not appearing, investigate why

Investigate if the aggregate GetById should return null

At the moment get by Async is never returning null, in fact it is using null! to mute the alert, but I think it should return null.

what to do if a consumer type is specified but no handlers are selected

inside the messsagehandler registry I should stop the code maybe? in case there is no handlers at all. search for #4 in the code

Service Discovery: Add tests

add tests for service discovery, only unt test, I have to prepare the environment for the integration tests

Improve the way the connection is retrieved in mongoDb

Atm we calculate the connection string in every request. Figure out the best way to administratte it; probably we can use memorycache for it.

It will be ideal also to have the entire provider stuff inide the mongodb project.

Try to improve also the way the databae is specified in the repositories, probably we can just specify IMongoDatabase in the constructor.

search by the issue in the code.

Create a nuget package out of the shared libraries, to facilitate the usage in other projects

It will be nice to have at least the consumers/Producers library as nuget package, because lot ofo companies ask for it on the take home interview.

Add tests for Distribt.Shared.Communication

Distribt.Shared.Communication has no test, Ideally it should contian both integration and unit test.

Create basic functionality for stock and warehouse microservices

in order to have a complete work on the products api we need to create a new service for the stock and another one for warehosues. atm the interface is faked.

RabbitMQ: Disposing Iconnection Imodel earlier than we should

When consuming messages we work with Iconnection and IModel both of them are IDisposable but inRabbitMQMessageConsumer.consume they were being disposed before the message was processed, which caused that we couldnt update the message to processed.

The current solution its not ideal, it works as long as the app does not scale crazy, but it has to be fixed.

API Gateway: update both apis to have the config in code

with the inclusion of the service discovery the URL definition (at least) should be in code rather than in the configuration.

Documentation: document distribt.shared.setup

investigate if an exception in the handlers breaks the entire consumer logic

When ConsumerHostedService gets executed, I need to investigate what happens if an exception jups in the handler.
Does it break the entire consumer process?

In addition some logs may be needed.

RabbitMQ: Implement dead letter queue

Refactor to use Rop

I started to use rop as an example in the code, it should be quite consistent the way is it done

or maybe not for clarity if someone else is checking the code; think about it.

FormatterServices is obsolete (inside the aggretage repository

FormatterServices is obsolete sincer net8, search a new way of implement this properly;
notes: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.serialization.serializationinfo.-ctor?view=net-8.0#system-runtime-serialization-serializationinfo-ctor(system-type-system-runtime-serialization-iformatterconverter-system-boolean)

search for the issue number on the code but it is in the aggregateRepository https://github.com/ElectNewt/Distribt/blob/main/src/Shared/Distribt.Shared.EventSourcing/AggregateRepository.cs#L39C44-L39C44

Documentation: add shared.Distribt.Discovery

RabbitMQ: messgereceiver locks the Thread

rabbitmq gets processed on a synchronous way, but the handler is asycn. for that reason we have to run the handler inside a task

var t = Task.Run(HandleMessage);
        t.Wait();

but this blocks the trhead, ideally if possible, this should be done in a different way.

Create shared.awaiter

in some scenarios (mainly config) we are executing async communication in non-async flows. for that reason instead of doing just .Result i should write a wrapper around it like Awaiter.Execute or something like that that executes that piece of code.

Secret manager: Add tests

both unit and integration.

Documentation for event sourcing and aggregates

there is lot of code in the eventsourcing project, it will be nice to add some documentation of how to use it. specially around the aggregate.

when using aggregates the classes have to be registered

when implementing event sourcing (and aggregates) with mongo, the classes that are "event" needs to be registered with BsonClassMap.RegisterClassMap<T>(); figure it out a way to not need this registration or do the registration abutomatically, as the events should be identificable by IApply<T>
Error Message: error occurred while deserializing the Content property of class Distribt.Shared.EventSourcing.AggregateChangeDto: Unknown discriminator value 'OrderCreated' where OrderCreated is T

Workarounds:

• Add all the events under a .RegisterClassMap
• store in the AggregateType the full type(namespace, etc) instead of only the name.

SecretManager: Add a wrapper to block endless calls

Investigate if it is needed to add a wrapper around the secret manager.
this wrapper will keep the secrets in memory and it will not call the service unless the call fails.

This is something to keep in mind, if we add the wrapper the consumers of the library will need to deal with the retry.

build an inmemory configuration for the healthchecks

All the health check configuration is normalized. will be nice to just to use an inmemory configuration in order to re/use the same and save some code.

of course, if there is something in the appsettings about the configuration it should be override.
this is the current configuration:

  "HealthChecksUI": {
    "HealthChecks": [
      {
        "Name": "Orders health check",
        "Uri": "/health"
      }
    ]
  }

update observability URL

ATM the observability is in the code hardcoded as localhost. move it to consul and get it from there instead.