runlyio/core-dotnet

Multi-threaded batch processing and background jobs for .NET Core

Multi-threaded batch processing and background jobs for .NET Core

Runly is a framework for simplifying complex applications by writing compartmentalized, testable code. You focus on writing your business logic and Runly gives you a CLI, multi-threading, retries, and more out-of-the-box.

How Does It Work?

You build your jobs as classes that inherit from Job into a new console app. You reference the Runly nuget package and you immediately have a fully functioning and robust CLI app.

🚀 Get Started

There are two ways to bootstrap a Runly job app:

1. Create a new job app from the GitHub template. This will create a new repository for you that you can clone and immediately start running.

2. Skip the Git repo and create a local app via the dotnet new template:

First, install the template:

dotnet new -i Runly.Templates

Then create a new project:

mkdir myproject
cd myproject
dotnet new runly-app

That's it - you can now start running! Learn more about how to build jobs in the Runly Quickstart.

Example

Here is a job that reads a CSV file (using CsvHelper) and sends an email to each person in the file.

public class CsvEmailer : Job<CsvConfig, Person>
{
    readonly MyEmailService emails;

    public CsvEmailer(CsvConfig config, MyEmailService emails)
        : base(config)
    {
        this.emails = emails;
    }

    public override IAsyncEnumerable<Person> GetItemsAsync()
    {
        // Return a list of items to process.
        // In this case, we will stream CSV records from a file.

        var csv = new CsvReader(File.Open(Config.CsvFilePath), CultureInfo.InvariantCulture);
        csv.Configuration.Delimiter = "|";

        return csv.GetRecordsAsync<Person>();
    }

    public override async Task<Result> ProcessAsync(Person person)
    {
        // Do the work to process each item.

        await emails.SendReminderEmail(person.Email, person.Name)
        return Result.Success();
    }
}

Using command line flags or JSON config, this can then easily be multi-threaded to send multiple emails at a time and allow for retries to deal with intermittent errors while sending the email.

See more in-depth examples or start reading the docs.

Why Write Code Like This?

A lot of problems that developers are tasked with solving everyday boil down to processing lists of items. Some examples could be:

• importing and processing records from large files into a database
• downloading or uploading import/export files on a schedule automatically
• queueing and sending large amounts of transactional emails
• generating PDFs from a list of records in a database
• handling ETL scenarios

A Runly job takes this pattern and standardizes it into a base Job class. Building and running jobs in a predictable way like this allows us to build lots of goodies on top of your core jobs. Things like multi-threading, scaling work across different machines, and resiliency and fault-tolerance become almost trivial for you to implement.

In fact, we have done exactly that and built a whole platform on top of this core OSS project.

The Runly Platform

You can turbocharge your job apps using the Runly Platform. The Runly Platform is SaaS which helps shepherd your jobs from deployment to monitoring and scaling in production.

• Automatically deploy your jobs to your existing machines or the cloud as NuGet packages using tools you already know and love.
• Use familiar REST concepts to queue and schedule background jobs from your web or mobile app.
• Easily integrate progress bars and job status into your app using Runly UI.
• Use CRON expressions for fine-grained control of job scheduling.
• Get insights into job failures and performance issues before your users notice.

You can get started on a generous free tier with no commitment and no credit card. Get started for free using your GitHub account or learn more about how it works.

Is this project sustainable?

Sustainability in OSS is a problem. We are working on this project full-time and in order to keep doing that, we have to feed our families. We think providing value via a paid service is the best way to do that while still solving problems for people with the core OSS offering.

We are fiercely independent and not going anywhere. Read more about our philosophy.

Our mission is to create great developer experiences by making it trivial to build and scale jobs in order to provide a great user experience in your app. We have plenty of ideas on how to do this but we need your support to continue working on this project.

You can support us by building a Runly job and letting us know what you think (tweet @ us or open an issue). If you find value in building Runly jobs, you can give the Runly Platform a try.

Or, just give us a GitHub star while you are here. It allows us to brag about all the internet points we are getting when our family asks what we are doing all day 😉.

You can keep up with what we are working on by following us on Twitter.

---

Consuming Pre-release Packages

Pre-release packages are published to this project's GitHub Packages Feed. If you want to consume one of these pre-release packages for testing purposes, you can add a nuget.config file to the root of your project:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="github" value="https://nuget.pkg.github.com/runlyio/index.json" />
    <add key="NuGet.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <github>
      <add key="Username" value="USERNAME" />
      <add key="ClearTextPassword" value="PERSONAL_ACCESS_TOKEN" />
    </github>
  </packageSourceCredentials>
</configuration>

This config file will instruct your nuget client to look for packages via Runly's pre-release feed first and then the official NuGet feed.

You will need to authenticate to the GitHub feed using a personal access token. Generate a new token with the read:packages scope and replace USERNAME and PERSONAL_ACCESS_TOKEN in nuget.config with your values.

core-dotnet's Issues

Write test case for bad process type in config file

When a process type cannot be found, throw an exception/alert the users to the issue.

Direct a run to a cluster or node

When queuing a run, the config should allow a cluster or node to be specified in order to direct the run to that cluster or node.

Share data between chained processes

When one process runs as a result of another process completing, match fields from the output of the first process with the config of the second and copy data (possibly only in the case the config does not have a value for a given field). E.g., if the process DataImport runs after MoveFile and MoveFile's output and DataImport's config have fields named "FilePath", then the value in MoveFile's FilePath would be set in DataImport's config. This allows for simple chaining together of processes where some data needs to be shared between the two.

Support setting config properties via command line

Similar to how the Node.js version works, it would be great for ease-of-use (and especially for quick demos) to be able to run a job without having to first create a json file first. Something like this:

myapp run helloworld --names Rick --names Morty --execution.parallelTaskCount 3

This would be the equivalent of this json config:

{
  "job": "helloworld",
  "names": ["Rick", "Morty"],
  "execution": {
    "parallelTaskCount": 3
  }
}

Jobs with no item type fail to run

Jobs that extend Job<TConfig> fail due to an invalid cast in SingleItemSource.

Update config with changes in core-nodejs

Update the donet config with changes made in the nodejs lib.

Integrate Konsole

Show progress bars and better formatting

When writing logs to the console, the formatting of the items processed and also the Press q to quit gets all janked up.

Konsole looks like a cool project that could help.

Job sticks around doing nothing after timing out on the server

To reproduce:

1. Queue a job and let it start.
2. Sever its connection to the API.
3. Wait for it to timeout on the server.
4. Restore the connection.

The job will be in an ambiguous state where the process is running and there is no feedback in the console.

Bug writing results to console

When an exception is thrown in Initialize:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Runly.ResultLog.ToString()
   at System.IO.TextWriter.WriteLine(Object value)
   at System.IO.TextWriter.SyncTextWriter.WriteLine(Object value)
   at System.Console.WriteLine(Object value)
   at Runly.Hosting.RunAction.RunAsync(CancellationToken token)
   at Runly.Hosting.RunAction.RunAsync(CancellationToken token)
   at Runly.System.Program.<Main>(String[] args)

Add integration testing library to be used by developers

A test harness that will run a process's initialize, process, etc. for integration-style testing.

Also guidance on how to use it and guidance on how to write unit-style tests without the harness.

Runly.Testing?