wuyichen24/spring-microservices-in-action

The source code of the book "Spring Microservices in Action (John Carnell)" and the personal summary of technical essentials about Spring Boot for microservices.

spring-microservices-in-action

This repository contains the source code of the book "Spring Microservices in Action (John Carnell)" and the personal summary of technical essentials about Spring Boot for microservices.

Overview

This source code was re-organized by the original source code of the book and I make sure each module is runnable.

Structure

• Servers
  • Config: Provide the configuration parameters for other services.
  • Eureka: Service discovery.
  • Zipkin: Collect tracing data.
  • Zuul: API gateway.
• Services
  • Licensing: Manage (CRUD) license records.
  • Organization: Manage (CRUD) organization records.
  • Authentication: Issue and validate access tokens, manage clients' and users' credentials and roles.
  • Special Routes: Provide an alternate endpoint for a certain service.
• Database: Store license and organization records, clients' and users' credentials, alternate routes of any service.
• Message Queue (Kafka): Notify the licensing service about any organization record modification.
• Cache (Redis): Cache any organization record which has been loaded before.
• Log Server
  • Splunk: Collect log information in local.
  • Papertrail: Collect log information in cloud.

Technology Stack

• Core
  • Spring
    • Spring Boot
    • Spring Data
      • Spring Data JPA
      • Spring Data Redis
    • Spring Cloud
      • Spring Cloud Config
      • Spring Cloud Stream
      • Spring Cloud Sleuth
    • Spring Security
      • Spring Security OAuth2
      • Spring Security JWT
      • Spring Security RSA
  • Netflix OSS
    • Netflix Eureka (service discovery)
    • Netflix Zuul (service gateway)
    • Netflix Hystrix (resiliency patterns)
    • Netflix Feign (HTTP API client)
    • Netflix Ribbon (service communication, load balancing)
• Message Queue
  • Apache Kafka with Apache ZooKeeper
• Cache
  • Redis
• Database
  • MySQL
• Security
  • OAuth2
• Log
  • Splunk
  • Papertrail
• Trace
  • Apache Zipkin
• Cluster Coordination
  • Apache ZooKeeper

Differences to The Original Source Code

• Use Gradle as Java build automation tool rather than Maven.
• Use MySQL as database rather than PostgreSQL.
• Change the port of the organization service to 8060 to avoid the address conflict with the licensing service.
• Change the port of the special routes service to 8040 to avoid the address conflict with the licensing service.
• Change the port of the authentication service to 8901 to make it same with the example in the book.
• Add the integration of Splunk for log aggregation locally (use both local Splunk and remote Papertrail).
• Use the original functionality in Logback for sending the log to the log servers (Splunk & Papertrail) rather than the solution in the Book (use Logspout to direct the docker output to the log servers).
• Replace the user_roles table by the authorities table for storing users' credentials to DB.
• Change the column name from user_name to username for the user_orgs table and the users table.
• Add comments to make the code easy to read.

Getting Started

• Gradle (Version >= 4.4)
• ZooKeeper
• Kafka
• Redis
• Splunk
• Command Memo

Technical Essentials

• Spring Basics
  • Autowired
• Data Model
  • SpringData JPA
• Configuration
  • Configuration File Auto-loading
  • Configuration Encryption
• Service Discovery
  • Service Discovery with Eureka
• Resiliency
  • Resiliency Patterns with Hystrix
  • Configure Hystrix
• API Gateway
  • Service Gateway with Zuul
  • Zuul Filters
• Security
  • Protect Service with Spring Security and OAuth2
  • Use JWT as Access Token
  • Store Clients and Users' Credentials to DB
• Stream
  • Integrate with Message Queue (Kafka)
• Cache
  • Integrate with Redis
• Logging
  • Tune Logging
  • Log Aggregation
• Tracing
  • Send Trace to Zipkin
• Deployment
  • Build Runnable Jar

Study Notes

• Google Doc

spring-microservices-in-action's Issues

Add audit for certain table

try to add audit for certain table: users, oauth_client_details, license, organization.

https://www.callicoder.com/hibernate-spring-boot-jpa-one-to-many-mapping-example/

references

• API 网关性能比较：NGINX vs. ZUUL vs. Spring Cloud Gateway vs. Linkerd
• 服务网关zuul初级篇
• Pattern: Backends For Frontends
• Spring Cloud Zuul网关 Filter、熔断、重试、高可用的使用方式

Dockerize each microservice module

Add database info for each service / server

Add license

add app in splunk (home laptop)

spring-microservices-in-action

Store trace data into other places

R&D for store trace data into other places:

• MySQL database
• Cassandra database
• Elasticsearch

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Send-Trace-to-Zipkin#places-for-storing-tracing-data

Create APIs for managing clients and users in authentication service

Create APIs for managing clients and users in the authentication service. like add, update, delete clients and users. and also encode passwords when add and update clients and users.

The API should be similar with licensing service and organization service for managing licenses and organizations.

references for log

https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html

Annotate a setter or a constructor by @Autowired

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Autowired#autowired-a-setter

Hibernate Mapping Annotation

• Annotations
  • OneToMany
  • ManyToOne
  • Enbedded
  • ElementCollection
• Link
  • https://www.callicoder.com/hibernate-spring-boot-jpa-one-to-many-mapping-example/
  • https://www.callicoder.com/hibernate-spring-boot-jpa-one-to-one-mapping-example/
  • https://docs.spring.io/spring-data/data-jpa/docs/current/reference/html/#jpa.query-methods

Send trace data into Zipkin server by other ways

R&D for sending tracing data into Zipkin server by other ways:

• Kafka
• RabbitMQ

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Send-Trace-to-Zipkin#setting-for-sending-trace-by-other-ways

Create a new internal library (application) to capture the same code about user context under util package

Effected modules

• authentication service
• authentication service jwt
• licensing service
• organization service
• specialroutes service
• zuul server

Files

• UserContext
• UserContextFilter
• UserContextHolder
• UserContextInterceptor

java.lang.IllegalArgumentException: Not a managed type: class com.thoughtmechanix.licenses.model.License

When you start a service, it will throw an exception:

Caused by: java.lang.IllegalArgumentException: Not a managed type: class com.thoughtmechanix.licenses.model.License
	at org.hibernate.jpa.internal.metamodel.MetamodelImpl.managedType(MetamodelImpl.java:210) ~[hibernate-entitymanager-5.0.12.Final.jar:5.0.12.Final]
	at org.springframework.data.jpa.repository.support.JpaMetamodelEntityInformation.<init>(JpaMetamodelEntityInformation.java:71) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.data.jpa.repository.support.JpaEntityInformationSupport.getEntityInformation(JpaEntityInformationSupport.java:68) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getEntityInformation(JpaRepositoryFactory.java:184) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getTargetRepository(JpaRepositoryFactory.java:117) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getTargetRepository(JpaRepositoryFactory.java:99) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:198) ~[spring-data-commons-1.13.18.RELEASE.jar:na]
	at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.initAndReturn(RepositoryFactoryBeanSupport.java:277) ~[spring-data-commons-1.13.18.RELEASE.jar:na]
	at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.afterPropertiesSet(RepositoryFactoryBeanSupport.java:263) ~[spring-data-commons-1.13.18.RELEASE.jar:na]
	at org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean.afterPropertiesSet(JpaRepositoryFactoryBean.java:101) ~[spring-data-jpa-1.11.18.RELEASE.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1689) ~[spring-beans-4.3.22.RELEASE.jar:4.3.22.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1627) ~[spring-beans-4.3.22.RELEASE.jar:4.3.22.RELEASE]
	... 42 common frames omitted

zipkin server send log to log server

Store the client application's credential in database

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Protect-Service-with-Spring-Security-and-OAuth2#store-in-database

Store the client users' credentials and roles in database or LDAP server

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Protect-Service-with-Spring-Security-and-OAuth2#store-in-database-1

Add a diagram to show the dependency relationship between each service.

Use short url for link

For directing to different section, change to (#section_name)

Use TCP traffic for log aggregation to Splunk

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Log-Aggregation#option-2-use-tcp-traffic

References

https://stackoverflow.com/questions/48296972/bean-creation-exception-with-org-springframework-security-config-annotation-web

java.lang.NoSuchMethodError: org.springframework.data.repository.config.AnnotationRepositoryConfigurationSource.<init>(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/Class;Lorg/springframework/core/io/ResourceLoader;Lorg/springframework/core/env/Environment;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V

When you start a service, it will throw an exception:

java.lang.NoSuchMethodError: org.springframework.data.repository.config.AnnotationRepositoryConfigurationSource.<init>(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/Class;Lorg/springframework/core/io/ResourceLoader;Lorg/springframework/core/env/Environment;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V
	at org.springframework.boot.autoconfigure.data.AbstractRepositoryConfigurationSourceSupport$1.<init>(AbstractRepositoryConfigurationSourceSupport.java:68) ~[spring-boot-autoconfigure-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.autoconfigure.data.AbstractRepositoryConfigurationSourceSupport.getConfigurationSource(AbstractRepositoryConfigurationSourceSupport.java:67) ~[spring-boot-autoconfigure-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.autoconfigure.data.AbstractRepositoryConfigurationSourceSupport.registerBeanDefinitions(AbstractRepositoryConfigurationSourceSupport.java:58) ~[spring-boot-autoconfigure-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsFromRegistrars(ConfigurationClassBeanDefinitionReader.java:360) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:144) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:116) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:320) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:228) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:272) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:92) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
	at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) [spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) [spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) [spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.14.RELEASE.jar:1.5.14.RELEASE]
	at com.thoughtmechanix.organization.Application.main(Application.java:28) [bin/:na]

Add a diagram to display the relationship between Zuul and eureka

Document the getting start and the integration with Papertrail

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Log-Aggregation#send-log-to-papertrail

References for build runnable jar for Spring boot

• https://stackoverflow.com/questions/43366015/gradle-build-with-fatjar-plugin-and-springboot-application-gives-application-st
• https://spring.io/guides/gs/spring-boot-docker/#initial
• https://guides.gradle.org/creating-multi-project-builds/#refactor_common_build_script_code
• https://blogs.oracle.com/fadevrel/useful-groovy-string-functions

full log aggregation

Not only all services send log to splunk, also let splunk track the log of Kafka, Zookeeper, Redis

Errors in book

• Page109: to see organization service is in registry, the url is "http://localhost:8761/eureka/apps/ORGANIZATIONSERVICE", not "http://localhost:8761/eureka/apps/organizationservice"
• Page242: configuration should be spring.cloud, not spring.application
• Page248: you need to add Content-Type: application/json in headers, otherwise it will give you the error "415 Unsupported Media Type" in response
• Page255: need to specify the client type as "rest" so that it will call the organization service with Redis logic.

Correct: http://localhost:8080/v1/organizations/e254f8c-c442-4ebe-a82a-e2fc1d1ff78a/licenses/f3831f8c-c338-4ebe-a82a-e2fc1d1ff78a/rest
Wrong: http://localhost:8080/v1/organizations/e254f8c-c442-4ebe-a82a-e2fc1d1ff78a/licenses/f3831f8c-c338-4ebe-a82a-e2fc1d1ff78a
(This will not get organization record and also will not trigger Redis logic)

Document API

API Documentation Solutions

• Swagger
• RAML
• API Blueprint
• Apiary

Extend the expiration of access token from authentication service

After restarting the authentication service, the access token will be changed.
Find a way to keep no change on access token if the authentication service is restarted.

Store Zipkin's tracing data into different holders

1. In-memory data
2. MySQL: http://mysql.com
3. Cassandra: http://cassandra.apache.org
4. Elasticsearch: http://elastic.co

Add javadoc on all java code

Finished module

• authentication-service
• authentication-service-jwt
• config-server
• eureka-server
• licensing-service
• organization-service
• specialroutes-service
• zipkin-server
• zuul-server

Readme error

Wrong description. Need to swap

Move the Zuul configuration into the config server

Page 168
Dynamically reload route configuration without restarting Zuul server.

How to pass the auth_token to downstream service which is requiring the access token

The organization service is protected. You can hit it only with an access token.

There are several cases for passing the access toke:

• http://localhost:5555/organizationservice/v1/organizations/e254f8c-c442-4ebe-a82a-e2fc1d1ff78a

The zuul server need to route the service.

• http://localhost:8080/v1/organizations/e254f8c-c442-4ebe-a82a-e2fc1d1ff78a/licenses/f3831f8c-c338-4ebe-a82a-e2fc1d1ff78a/rest

The licensing service needs to hit the organization service to retrieve relevant organization data.

References for Zipkin

https://cloud.spring.io/spring-cloud-sleuth/1.2.x/multi/multi__span_data_as_messages.html

Request for instructions to start the application

Hi ,

I did not find any instructions on how to build and start the application and test the application which is deployed on localhost environment. can you please kindly create a document which explain the details.

Thanks,
Herambha Kumar P

Load the application configuration file in YAML format

https://github.com/wuyichen24/spring-microservices-in-action/wiki/Configuration-File-Auto-loading#load-the-application-configuration-file-in-yaml-format