rajibmitra/coolretailer

Microservices with Istio, gRPC, Redis, BigQuery, Spring Boot, Spring Cloud and Stackdriver

CoolRetailer

Microservices application with Istio, gRPC, Redis, BigQuery, Spring Boot, Spring Cloud and Stackdriver

• Smart product finder with autocomplete feature
• Distributed Stackdriver Tracing with Log correlation across gRPC calls
• Istio based Stackdriver Service Monitoring
• Automatic synthetic load generation with Locust
• Complete CI/CD to GKE with Google Cloud Build
• Redis in Master/Slave replication mode (image from Bitnami)
• Autocomplete logic based on ZRANGEBYLEX Redis command
• Tunable cache size
• Dataset from BestBuy (converted to ndjson using JsonProcessor)
• Developed completely with VSCode
• Inspired by Next '18 Microservices Demo
• Covering slides on Slideshare

Architecture:

Automatic Installation:

Guided one click installation from Google Cloud Shell. Complete stack (incl. all dashboards shown below + Istio fault simulation). No client tooling required.

Manual Installation:

• Get the Best Buy products dataset and save it to a GCS bucket:
• BQ requires the JSON to be new line delimited. Use the provided utility JSON processor for this:

java -jar src/queryservice/target/queryservice-1.0.0.jar \
    --spring.profiles.active=JSON \
    --input.json="$PWD"/products.json --exit

• Load it to BQ as dataset coolretailer.products from GCS
• Create a GCP service account with the following roles:

BigQuery Data Viewer, BigQuery Job User, Stackdriver Debugger Agent, Cloud Trace Agent, Context Graph Asserter, Error Reporting Admin, Logs Writer, Monitoring Metric Writer

and install the private key as app-gac.json as kubernetes secret app-gac.

• apply Istio manifests in: setup/istio-manifests
• set PROJECT_ID, TAG_NAME, REDIS_HOST in setup/kubernetes-manifests/ and apply them

Notes:

• Spring Cloud Sleuth Stackdriver Trace and Logging enabled.
• Complete CI/CD pipeline with Google Cloud Build.
• Test locally using:

$mvn spring-boot:run \
    -DGOOGLE_APPLICATION_CREDENTIALS=path-to-key-file \
    -DPROJECT_ID=coolretailer \
    -DGOOGLE_CLOUD_PROJECT=coolretailer \
    -Dspring-boot.run.arguments=--spring.redis.host=localhost

• Using docker from Cloud Shell:

(If tunneling to the redis service is required)

gcloud container clusters get-credentials coolretailer2 --zone europe-west4-a --project coolretailer \
 && kubectl port-forward $(kubectl get pod --selector="app=redis-master,role=master,tier=backend" --output jsonpath='{.items[0].metadata.name}') 6379:6379

gcloud auth configure-docker -q &&\
docker run \
    --network=host \
    -v /home/kbhattacharya:/etc/app-gac \
    -e GOOGLE_CLOUD_PROJECT=coolretailer \
    -e PROJECT_ID=coolretailer \
    -e GOOGLE_APPLICATION_CREDENTIALS=/etc/app-gac/app-gac.json \
    -e --spring.sleuth.sampler.probability=1.0 \
    -e --spring.application.name=ux-local \
    -e --spring.cloud.gcp.trace.enabled=true \
    -e --spring.cloud.gcp.logging.enabled=true \
    -e --spring.cloud.gcp.project-id=coolretailer \
    -it gcr.io/coolretailer/queryservice

• Use Cloud Memorystore instead of the Redis deployments (src/setup/kubernetes-manifests/archive/redis):
• Istio TLS setup
• See setup/istio-manifests/secure for simple TLS configuration
• The static content is recommended to be served from a GCS bucket with Cloud CDN. Just drop off the content of src/ui/static/ui.* and src/ui/static/lib in a GCS bucket and provide allUsers READER access.

Screenshots:

Application

Cloud Trace with Log correlation

Istio fault injection

Grafana

Istio Service monitoring

Kubernetes CPU telemetry

Disclaimer:

This is not an official Google product