differentialhq/differential

Typescript-first background services platform with durable functions.

differential's Issues

Add a fly deploy step circumventing tests

We need a fly deploy step as an action to "break-glass" and proceed with deployments when the pre-deploy tests fail.

Caching test fails intermittently

See the run here: https://github.com/differentialhq/differential/actions/runs/7762851286/job/21174002166?pr=102

Needs to be investigated whether this is symptomatic of a bigger issue, or a stale-read problem.

When starting a service, shut it down on a possible 401 with an exit code 1

A small critic on the license

In 2024 it's not hard to create a E2E system with Node, TS and some client-side fetch. Elysia and Hono, the newest frameworks, already support this use case. I think your monetization approach by giving the open source project a viral license is flawed, as it will just make people look the integrations you did with background queues and other extra batteries and they will do it by hand on the newer frameworks.

Decision: Websockets and persistent connections

Differential implements a very chatty protocol. This will get to be a problem as and when we scale.

However, implementing websockets and persistent connections has it's own challenges. Whenever you say the connection is persistent, you run into the problem of the server maintaining state of that connection. And that adds extra complexity, because you can't make the same assumptions as you'd have made when the server was stateless.

For example, you can't scale down (easily) without kicking off clients. And you can't scale up (easily) and rebalance as you wish. You can implement/buy a proxy that handles these concerns for you, but that's additional complexity.

Instead, we will stick with the tried and true long polling for most of our chatty workloads with keep-alives and HTTP2 support.

We may visit going fully websockets in the future, but I've a feeling we don't need to.

Handling client abort

In certain circumstances, the polling agent might close the connection before getting the request, but after the atomic UPDATE has marked the job as running. In these cases, we should try to walk back the update by listening to the abort.

Some Fastify references:

• fastify/fastify#4518
• fastify/help#577