eloots/pi-akka-cluster

Raspberry Pi - Kubernetes Cluster

Description

This code is used on a Raspberry-Pi based Akka Cluster to visually demonstrate cluster formation, split-brain occurrence and Split Brain Resolver.

Each node is equipped with an 10-LED RGB strip where different LED's are used to show each node's state (Joining, Up, Weakly-up, Unreachable, Leaving, Exiting and Down/Removed). In addition, it shows if a node has a so-called leader role and whether a node is running an Akka Cluster Singleton (if one is created on the cluster).

In addition to 10-LED Cluster there is also OLED display based cluster. Instructions for display assembly can be found here

In the last series of exercises, we deploy an actor based application (a Sudoku Solver) on the cluster and add monitoring (Cinnamon/Prometheus/Grafana) and tracing to the set-up (Open-tracing/Zipkin).

Maintainers & Changelog

Removed the changelog & list of maintainers. Git history contains this information.

Instructions

• Make sure you completed the installation of
  • Java 11 SDK.
  • Some terminal (iTerm-2 on Mac, Terminator on Linux style OS-es, Powershell or ConEmu on Windows).
  • curl.
• Clone this project on your laptop.

Preparing your laptop for a Raspberry-Pi based Akka Cluster workshop

Detailed instructions can be found here.

Building your own 5-node Raspberry-Pi based Akka Cluster

If you want to build your own physical cluster, you may want to have a look at the BOM here.

Pi-Hypriot installation instructions

The easiest and fastest way to flash a Hypriot OS instance that is ready to use for the Akka Cluster software can be found here.

Display version instructions are a little different and available here

Travel router configuration

In some cases it can be handy to use a so-called travel router which allows you to connect to a Wifi network on location, and still get access to a physical network with a fixed IP network address. This is because, currently, all nodes have statically configured IP addresses.

Details on how to configure such a (TP-Link TL-WR802N) router can be found here.

Build your own Akka-Cluster on Raspberry Pi

• A bill of material can be found here
• A custom 10-LED strip that can be directly plugged on a Raspberry Pi board's GPIO connector can now be bought from Lunatech! You can buy them in a pack of 5. For more info, send a mail to [email protected].
• A schematic of the LED strip can be found here

Instructions for Display assembly are available here

Running 64-bit Ubuntu

Ubuntu has a 64-bit ARM version with built-in cloud-init support. As such, you can flash 64-bit capable boards (Raspberry Pi 3 or 4) with this operating system. The akka-pi-os-dhcp-64.yml cloud-init file has the required modifications to flash a card for that version of the OS on ARM, with the most important difference with the akka-pi-os-dhcp-32.yml being a 64-bit binary for the installation progress tracker.

This has been verified on a RPi4. A slightly modified version of the Hypriot flash utility is required though.

flash -n node-1 -u akka-pi-os-dhcp-64.yml \\n   http://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz

pi-akka-cluster's Issues

Server Change

Question regarding other server hardware - resolved.

Unused variable

(Just an example to illustrate the point I made in the review of PR #151)

As reported by the linter, variable serverSource is never used.

https://github.com/lightbend/Pi-Akka-Cluster/blob/26b4a519ffd8211e842f28e8aff97731efc48ce8/exercise_102_display_cluster_sharding/src/main/scala/com/lightbend/akka_oled/Main.scala#L69

Capitalisation of settings

Currently, most of the names of settings (in the Settings object) do not have their first letter capitalised. This should change for two reasons:

• Better for reading/inspecting or reviewing the code that use such settings.
• Allows to match on settings as a constant in pattern matching and avoids the pitfall that you get a unwanted variable pattern match.

Update Exercise 50 readme file for instructions

Exercise readme needs to be updated to include instructions describing the exercise and what to do to run it.

Move deployment/run logic to sbt

Basically what I'm seeing is to have "sbt run" and this will do assembly, copy jar to nodes and start app across all nodes.
Perhaps "sbt run 0" would be helpful as well
I see we should add:

• deploy
• run
• stop
• shutdown
• copyLogs
• showLogTail

For this case, "run" means opening ssh session and run app in background with console output to file (with ability to tail -f from this file).
Still, node located run script is still valuable as it allows us to see logs right away so perhaps this should stay.
CC: @eloots @kikiya

Akka-pi-OS-install-instructions.md dont seem to work

Pi-Akka-Cluster/Akka-Pi-OS-install-instructions.md dont seem to work

flashing that image to the SD gets me stuck with this screen on the rasberry pi

PR #130 breaks existing exercises

@agolubev
PR #130 breaks existing exercises. E.g. running exercise 60. Building runs fine (universal:packageBin). However, running the exercise on a node produces the following error:

$ ./run 60
Running exercise_060_cluster_base on node-1
10:18:49 INFO  [] - Slf4jLogger started
SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
10:18:49 INFO  [ArteryTcpTransport(akka://pi-cluster-0-system)] - Remoting started with transport [Artery tcp]; listening on address [akka://pi-cluster-0-system@node-1:2550] with UID [3795535384498228207]
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Starting up, Akka version [2.6.1] ...
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Registered cluster JMX MBean [akka:type=Cluster]
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Started up successfully
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing
10:18:50 WARN  [ManifestInfo(akka://pi-cluster-0-system)] - Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.6.1] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.6.1, 2.5.23] in libraries [akka-discovery:2.6.1, akka-coordination:2.6.1, akka-actor:2.6.1, akka-remote:2.6.1, akka-cluster-sharding-typed:2.6.1, akka-cluster:2.6.1, akka-protobuf-v3:2.6.1, akka-distributed-data:2.6.1, akka-cluster-typed:2.6.1, akka-cluster-tools:2.6.1, akka-persistence:2.6.1, akka-cluster-sharding:2.6.1, akka-actor-typed:2.6.1, akka-persistence-query:2.5.23, akka-slf4j:2.6.1, akka-persistence-typed:2.6.1, akka-stream:2.6.1]
Exception in thread "main" java.lang.IllegalStateException: Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.6.1] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.6.1, 2.5.23] in libraries [akka-discovery:2.6.1, akka-coordination:2.6.1, akka-actor:2.6.1, akka-remote:2.6.1, akka-cluster-sharding-typed:2.6.1, akka-cluster:2.6.1, akka-protobuf-v3:2.6.1, akka-distributed-data:2.6.1, akka-cluster-typed:2.6.1, akka-cluster-tools:2.6.1, akka-persistence:2.6.1, akka-cluster-sharding:2.6.1, akka-actor-typed:2.6.1, akka-persistence-query:2.5.23, akka-slf4j:2.6.1, akka-persistence-typed:2.6.1, akka-stream:2.6.1]
	at akka.util.ManifestInfo.checkSameVersion(ManifestInfo.scala:206)
	at akka.util.ManifestInfo.checkSameVersion(ManifestInfo.scala:173)
	at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:1040)
	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:1029)
	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:1029)
	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:1052)
	at akka.actor.typed.ActorSystem$.createInternal(ActorSystem.scala:278)
	at akka.actor.typed.ActorSystem$.apply(ActorSystem.scala:192)
	at akka.cluster.pi.ClusterStatusTrackerMain$.main(ClusterStatusTrackerMain.scala:46)
	at akka.cluster.pi.ClusterStatusTrackerMain.main(ClusterStatusTrackerMain.scala)
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Exiting completed
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Shutting down...
10:18:50 INFO  [Cluster(akka://pi-cluster-0-system)] - Cluster Node [akka://pi-cluster-0-system@node-1:2550] - Successfully shut down
10:18:52 INFO  [akka://pi-cluster-0-system@node-1:2550/system/remoting-terminator] - Shutting down remote daemon.
10:18:52 INFO  [akka://pi-cluster-0-system@node-1:2550/system/remoting-terminator] - Remote daemon shut down; proceeding with flushing remote transports.
10:18:53 INFO  [akka://pi-cluster-0-system@node-1:2550/system/remoting-terminator] - Remoting shut down.
HypriotOS/armv7: akkapi@node-1 in ~
$

I did a short investigation and it seems that the problem is caused by some incompatibilities in the code you added in the common project. When the code that was added in common is removed, the exercise is rebuilt and run, the exercise runs fine. It seems that some incompatibilities exist between the current dependencies and the ones that you added.

Can you please fix asap?
Also, it is vital to check that existing exercises continue to function when applying changes. Please run tests on a cluster that allows testing this.

Add info on how to physically build cluster

Detailed instructions on:

• building the LED strip GPIO header assembly
• add instructions on how to build, copy and run exercises (stages)

Add support for upcoming custom 10-LED strip

With the prototype of the custom-designed having been verified to work (and the production of a first batch started), some changes need to be made to support this new strip (and retain support for the existing 8-LED strips). The LED order is reversed on the new strip and it has 2 extra LEDs.

• Add support for 10-LED strip with reversed LED order
• Provide an easy way to select the type of strip used on a cluster

Compilation error when moving to Scalatest 3.1.0

@agolubev
I did a short test with changing from ScalaTest 3.0.8 to 3.1.0.

Apart from the fact that there are deprecation warning (e.g. on WordSpec and Matcher), there are compilation errors related to code in eroled.SmartOLEDTest:

man [e] > Pi-Akka-Cluster > initial state > test
[info] Compiling 1 Scala source to /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/target/scala-2.13/test-classes ...
[error] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:7:12: object scalatestplus is not a member of package org
[error] import org.scalatestplus.mockito.MockitoSugar
[error]            ^
[error] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:9:43: not found: type MockitoSugar
[error] class SmartOLEDTest extends WordSpec with MockitoSugar {
[error]                                           ^
[error] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:12:33: type mismatch;
[error]  found   : [T](x$1: Class[T], x$2: org.mockito.MockSettings)T <and> (x$1: Class[T], x$2: org.mockito.stubbing.Answer[_])T <and> (x$1: Class[T], x$2: String)T <and> (x$1: Class[T])T
[error]  required: eroled.SmartOLED
[error]          val a: SmartOLED = mock[SmartOLED]
[error]                                 ^
[error] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:26:33: type mismatch;
[error]  found   : [T](x$1: Class[T], x$2: org.mockito.MockSettings)T <and> (x$1: Class[T], x$2: org.mockito.stubbing.Answer[_])T <and> (x$1: Class[T], x$2: String)T <and> (x$1: Class[T])T
[error]  required: eroled.SmartOLED
[error]          val a: SmartOLED = mock[SmartOLED]
[error]                                 ^
[error] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:46:33: type mismatch;
[error]  found   : [T](x$1: Class[T], x$2: org.mockito.MockSettings)T <and> (x$1: Class[T], x$2: org.mockito.stubbing.Answer[_])T <and> (x$1: Class[T], x$2: String)T <and> (x$1: Class[T])T
[error]  required: eroled.SmartOLED
[error]          val a: SmartOLED = mock[SmartOLED]
[error]                                 ^
[warn] /Users/ericloots/Trainingen/LightbendTraining/Pi-Akka-Cluster/common/src/test/scala/eroled/SmartOLEDTest.scala:7:34: Unused import
[warn] import org.scalatestplus.mockito.MockitoSugar

Bumping powermock-api-mockito2 from 2.0.2 to 2.0.4 doesn't resolve the issue.

Would be great to figure out whats' happening and fix this when a solution is available so that we can move to ScalaTest 3.1.0

Move Cluster Singleton indicator settings to configuration

In this version, as part of a quick demonstration, the LED number & color to indicate that a cluster singleton is running on a node is hard-coded. This should move to configuration

Add new exercise on 'Down All' Split brain resolver strategy

Recently, a new Akka Cluster Split Brain Resolver strategy, "Down all" was added. There should be an exercise to demo/try it.

librpi_ws281x works with 32Bit only

Hi,

I've tried to run the first exercise and the node complained with the following exception. My cluster setup is roughly the same besides the OS. I'm using the 64Bit version of Ubuntu for the RaspberryPi, I think thats causing the problem. So, the solution is probably to recompile the librpi_ws281x.so on a 64Bit Pi, but I have not worked with native libraries. Therefore it would be very helpful if you could provide instructions on how to recompile the library.

/home/akkapi/exercises/exercise_000_initial_state-1.3.0/lib/librpi_ws281x.so: wrong ELF class: ELFCLASS32 (Possible cause: architecture word width mismatch)
        at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
        at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
        at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
        at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2659)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.base/java.lang.System.loadLibrary(System.java:1873)
        at org.neopixel.Main$.main(Main.scala:27)
        at org.neopixel.Main.main(Main.scala)

Update Akka Management to first major release

Akka Management 1.0.0 was released recently. This project should use this version (and bump other dependencies as required).

sbt version was updated incorrectly

PR #130 changed sbt version from 1.3.5 to 1.3.3. Please revert this change.

Broken link in display-version.md

The link to the display installation instructions at the bottom of display-version.md is broken.

This is because there is a typo in the name of the actual .md file (the 'r' in instructions is missing): https://github.com/eloots/Pi-Akka-Cluster/blob/master/docs/display/display-installation-instuctions.md and the link is spelt correctly: https://github.com/eloots/Pi-Akka-Cluster/blob/master/docs/display/display-installation-instructions.md

The solution is either:

• Correct the spelling of the actual display-installation-instuctions.md file, or;
• Mis-spell the link at the bottom of display-version.md

SD card to BOM

On the BOM we should also add SD cards

/Pi-Akka-Cluster/images/BOM.md

Add support for Raspberry Pi 3 Model B+

Thus far, all deployments have been done using Raspberry Pi 3 Model B boards. With the arrival of the B+ version of this board, we should try to add support for it.
The current, fully configured Hypriot based OS distribution for this project doesn't run on the B+. As the Hypriot OS now supports the B+ we should be able to build a new image that works.

Continuous Integration testing?

It would be cool is we could at least run one or two scenario's in CI - perhaps using Travis and docker?

Warn user that 'unzip' should be installed on nodes when using native packager based scripts

Clean up project to remove unused exercises and re-number

• Remove unused projects and renumber

Remove vizceral project?

Should we remove the vizceral project since it's no longer supported by telemetry team and also I believe deprecated by netflix as well? @eloots

Use Artery TCP

If you are not using it already I'd recommend that you switch to Artery TCP that is new in Akka 2.5.11. It would be some additional indirect testing of the transport that will probably become the future (in 2.6) default. Thanks

See config in https://doc.akka.io/docs/akka/current/remoting-artery.html

Link markdown broken in display-installation-instuctions.md

IN this file, display-installation-instuctions.md, the markdown for one link is broken and the text appears thusly:

As next step you need to configure password-less login. Procedure is described [here](../Hypriot-OS-Course-Preparation-Instructions.md#Configure password-less login)

The relative URL path (to Hypriot-OS-Course-Preparation-Instructions.md) is valid, but the markdown of the link is broken, such that the link is not clickable.

Fix separated-out LED status monitor

The base code for the separated-out cluster status indicator is present but needs to be updated to:

• Display Weakly-Up nodes using a blinking LED
• Make it consistent with the bundled status monitor

Also, choosing between the bundled or the separated-out monitor should be done via configuration

Add exercises on Akka Cluster Bootstrap/Discovery

• Discovery via config already available in PR #83
• Discovery via Akka DNS

Move ClusterStatusMonitor to a separate package

The cluster status monitor code currently is in package org.neopixel. This should be moved across all exercises to a separate package akka.cluster.pi (as done in #124 for exercise 2).