roma-glushko/migdalor

🔆 A Kubernetes native peer discovery and cluster membership management library for your modern Python microservices

Migdalor

Migdalor is a cluster membership library for modern asyncio Python distributed systems running in Kubernetes.

Migdalor doesn't require a separate broker (e.g. Redis, etcd, Zookeeper, Chubby, etc) to work, but leverage Kubernetes out-of-the-box capabilities to solve the peer discovery problem.

Features

• 🐍 Modern Asyncio Pythonic API
• 🔦 Brokerless Kubernetes native peer discovery based on headless services
• 🔭 Hooks into membership change events
• 🛠️ Ability to manage membership manually

Installation

pip install midgalor
# or
# poetry add midgalor
# pdm add midgalor

Usage

import migdalor

cluster = migdalor.Cluster(
    node_address=(node_address),  # the current node address (e.g. 127.0.0.1:8001)
    discovery=migdalor.KubernetesServiceDiscovery(service_address=cluster_address), # Kubernetes headless service address (e.g. cluster:8000)
    ## Callbacks on different events
    # nodes_added_handlers=[...] 
    # nodes_removed_handlers=[...],
    ## Membership update rate
    # update_every_secs=10,
)

await cluster.start()

# You can also add or remove nodes manually if you support that in your protocol
await cluster.add([("127.0.0.1", 8001)])
# await cluster.remove([("127.0.0.1", 8001)])

await cluster.stop()

Midgalor comes with some comprehensive example to help you get started:

• The Party Cluster - An example of using Migdalor to implement peer discovery in Kubernetes cluster

migdalor's Issues

Implement IP caching on DNS failures

Allow to cache membership info on DNS failures and implement some simple recovery strategy (e.g. shorten the next update time significantly to retry discovery that way and then grow that delay exponentially).

Cover codebase by unit tests