ewolff/microservice-kafka

Demo to show how Apache Kafka can be used for communication between microservices

microservice-kafka's Issues

Proxy error

After building with maven and the docker build -> docker compose up -d

invoicing and shipping exit with code 1 and the call to localhost:8080 - clicking on order results in:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: DNS lookup failure for: order

Apache/2.4.18 (Ubuntu) Server at localhost Port 80

Question: use of `-XX:TieredStopAtLevel=1 -noverify`

Hello. I see that the Docker starts things with jvm flags -XX:TieredStopAtLevel=1 -noverify I can see how that might help with startup time and possibly memory. Seems that folks might copy those settings and I am wondering if they are recommended for production or simply there for fast demos?

Request without post or get

Hi @ewolff ,

We followed your recommendations from this issue and found a few instances of request method without get or post

Source:

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/main/java/com/ewolff/microservice/shipping/web/ShippingController.java

Line 28 in 918d538

@RequestMapping("/")

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/main/java/com/ewolff/microservice/order/logic/OrderController.java

Line 45 in 918d538

@RequestMapping("/")

and

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/main/java/com/ewolff/microservice/invoicing/web/InvoiceController.java

Line 28 in 918d538

@RequestMapping("/")

mvn package build issue

I am getting this issue with the build:

2017-08-23 09:00:25.822 ERROR 11897 --- [           main] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='{  "id" : 1,  "customer" : {    "customerId" : 1,    "name" : "Wolff",    "firstname" : "Eberhard", ...' to topic order:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

I use Linux Mint (ubuntu based distro). Everything is pretty up to date. Maybe a port is in use, or I am missing something? Or is it the 'null' that is a problem?

I tried a fresh build with mvn -X package and redirected the output. Could not see anything obvious. Maybe you can try and rebuild it and see if anything has changed?

Passwords in `.properties` file

Hi @ewolff ,

Our tool detected some instances of passwords in .properties file. Wanted to bring it to your attention:

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/test/resources/application-test.properties

Line 4 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/main/resources/application.properties

Line 13 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/test/resources/application-test.properties

Line 6 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/main/resources/application.properties

Line 11 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/test/resources/application-test.properties

Line 4 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/main/resources/application.properties

Line 13 in 918d538

spring.datasource.password=dbpass

returned a non-zero code: 137

I keep getting this error when trying to docker-compose build

Is there any issue with the file /apache/Dockerfile ?

ERROR: Service 'apache' failed to build: The command '/bin/sh -c apt-get install -y -qq apache2 && a2enmod proxy proxy_http proxy_ajp rewrite deflate headers proxy_connect proxy_html lbmethod_byrequests && mkdir /var/lock/apache2 && mkdir /var/run/apache2' returned a non-zero code: 137

Docker version 19.03.12, build 48a66213fe

Postgres init-user-db.sh not executed when using docker on Windows

The postgres container stopped prematurely with this error message in the log:
/usr/local/bin/docker-entrypoint.sh: line 170: /docker-entrypoint-initdb.d/init-user-db.sh: cannot execute: required file not found

The message is misleading, because the file really existed. Root cause; Bash cannot process files with CR/LF line endings.

Please add docker/.gitattibutes containing * text eol=lf or advice to set git config --global core.autocrlf false before cloning the repo.

BTW: I enjoyed reading "Microservices Praxisbuch".

Failed to execute ms-kafka-order

If the error shows,
Fatal error compiling: invalid target release: 10 -> [Help 1] mac
Either:

1. change the pom java properties version from 10 to the version 8 (as my system has java 8)
2. Update Java in local from your system java version to version 10.

Race condition of topic creation from KAFKA_CREATE_TOPICS & the Spring apps (?)

I was playing around with the demo, adding a third service and trying to scale the Spring app consumers and encountered the issue of for some reason getting stuck at having one partition, in spite of not modifying the docker-compose.yaml file KAFKA_CREATE_TOPICS from the original

Topic: order    PartitionCount: 1       ReplicationFactor: 1    Configs: 
        Topic: order    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

Though when I ran the app for the first time, I had no problems having the partitions be created correctly (and then scaling the consumers). I did some searching and found this issue, which seems to have the same issue: wurstmeister/kafka-docker#490

However the proposed solution there did not fix the issue for me in and of itself. I then found this topic wurstmeister/kafka-docker#569 which gave me a solution: disabling KAFKA_AUTO_CREATE_TOPICS_ENABLE. This solution seems to have permanently fixed the partition count issue for me.

(disclaimer: first time working with Kafka)

EDIT: I had some bugs in my extra Spring microservice code concerning json serialisation, without any particular startup error logs regarding this. Poking around in the app after the fix #24 I noticed this. Maybe probably that was the cause for allowing the race condition to occur.

mvn package installation should be done via docker multi-stage build

https://docs.docker.com/develop/develop-images/multistage-build/

Invoice and Shipment services don't start due to missing topic ?

tried to run the application and i am able to create / submit an order - however the invoice and shipment services are not running. when checking the docker log files for those services i get the following hint :

2020-07-04 15:22:34.933 INFO 6 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1593876154931
2020-07-04 15:22:35.134 WARN 6 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: Topic(s) [order] is/are not present and missingTopicsFatal is true
2020-07-04 15:22:35.135 WARN 6 --- [ main] o.s.b.f.support.DisposableBeanAdapter : Invocation of destroy method failed on bean with name 'tomcatMetricsBinder': java.lang.NullPointerException
2020-07-04 15:22:35.136 INFO 6 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
2020-07-04 15:22:35.136 INFO 6 --- [ main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2020-07-04 15:22:35.137 INFO 6 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2020-07-04 15:22:35.144 INFO 6 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2020-07-04 15:22:35.146 INFO 6 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-07-04 15:22:35.157 INFO 6 --- [ main] ConditionEvaluationReportLoggingListener :

I want to change db from postgresql to mysql.

Is there any way to change postgres db to mysql. I was trying to do so but it's not working, can you help me out for this.

How does Kafka Converts Order info to Invoice

Hello,
I was going through the code and trying to understand it. In OrderService class, method fireOrderCreatedEvent has kafkaTemplate.send("order", order.getId() + "created", order); statement, which sends Order object to Kafka messaging system.

private void fireOrderCreatedEvent(Order order) {
		kafkaTemplate.send("order", order.getId() + "created", order);
	}

And in OrderKafkaListener class you have the following method

@KafkaListener(topics = "order")
	public void order(Invoice invoice, Acknowledgment acknowledgment) {
		log.info("Revceived invoice " + invoice.getId());
		invoiceService.generateInvoice(invoice);
		acknowledgment.acknowledge();
	}

How did Order object serialize or converted to Invoice object? I am kind of confused here. Does Kafka implement this? or we need to manually do it here.

Starting mskafka_zookeeper_1 ... error ERROR: for mskafka_zookeeper_1 Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error ERROR: for zookeeper Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error ERROR: Encountered errors while bringing up the project.

D:\Raj_Codings\microservice-kafka-master\docker>docker-compose up -d
mskafka_postgres_1 is up-to-date
Starting mskafka_zookeeper_1 ... error

ERROR: for mskafka_zookeeper_1 Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error

ERROR: for zookeeper Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error
ERROR: Encountered errors while bringing up the project.

D:\Raj_Codings\microservice-kafka-master\docker>docker-machine create --virtualbox-memory "4096" --driver virtualbox dev
Running pre-create checks...
Error with pre-create check: "VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path"

docker compose fails

when trying out the docker compose section, I get this failure:

Building order
Step 1/4 : FROM openjdk:11.0.2-jre-slim
11.0.2-jre-slim: Pulling from library/openjdk
27833a3ba0a5: Pull complete
16d944e3d00d: Pull complete
6aaf465b8930: Pull complete
0684138f4cb6: Pull complete
67c4e741e688: Pull complete
783178df3a0c: Pull complete
Digest: sha256:da5605d071e9e41803f8b45f41f8c24ad6c26056190a871c8d6471982407d05b
Status: Downloaded newer image for openjdk:11.0.2-jre-slim
---> b7a931ed7d37
Step 2/4 : COPY target/microservice-kafka-order-0.0.1-SNAPSHOT.jar .
ERROR: Service 'order' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder600758216/target/microservice-kafka-order-0.0.1-SNAPSHOT.jar: no such file or directory

project builds, but tests fail

Hi, I wanted to try out your project, so cloned and tried out the how-to-run section. When the tests run after the build, there is a long period of getting these messages:

2019-09-30 19:51:25.888 WARN 88574 --- [ main] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=test] Connection to node -1 could not be established. Broker may not be available.

When the tests do end, this is the final output:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.234 s - in com.ewolff.microservice.order.kafka.OrderKafkaTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] OrderServiceTest.lastCreatedIsUpdated » IllegalState Failed to load Applicatio...
[INFO]
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] microservice-kafka ................................. SUCCESS [ 0.558 s]
[INFO] microservice-kafka-order ........................... FAILURE [02:19 min]
[INFO] microservice-kafka-shipping ........................ SKIPPED
[INFO] microservice-kafka-invoicing ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:20 min
[INFO] Finished at: 2019-09-30T19:43:11-07:00
[INFO] Final Memory: 42M/157M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project microservice-kafka-order: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/dstadler/Documents/code/k8s-mcsvs-java/microservice-kafka/microservice-kafka/microservice-kafka-order/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :microservice-kafka-order
C02Z65TVLVDQ:microservice-kafka dstadler$

Oders are not processed

When I create an order it takes quite some time before I get the success message, but it seems the messaging is broken since I don't see any invoice or shipment being created.

If there are any logs I could provide I'd need some help to locate them since I am a newby when it comes to web/backend development.