vangav/vos_backend

vangav open source - backend; a backend generator (generates more than 90% of the code needed for big scale backend services)

on YouTube

vangav backend

built on top tech

tech	ref
cassandra	apple's cassandra deployment has over 75,000 nodes storing over 10PB of data
play framework	play framework powers linkedin's 500,000,000 members backend
akka	akka helps power e-shopping for hundreds of millions of amazon members
datastax drivers	datastax processes over 1,000,000,000,000 cassandra requests for netflix per day

fully-loaded

utility	options
backend generator	design in json and generate 90+% of the code for a service + several utilities
security	facebook/google auth, oauth 2, transaction tokens, encryption, ...
networks	twilio, mail gun, rest client, rest jobs, ...
push notifications	ios, android, ...
geo services	geo grids, reverse geo coding, geo hashing, ...
dispatcher/worker	dispatchable queries, push notifications, emails, sms, ...
and more ...	periodic jobs, thread pools, facebook graph api, snow flake ids (twitter's ids), compression, client generator, exceptions, geometry, metrics, deep vangav mighty integration, ...

great templates

template	services	features	db tables	code: total	code: generated
instagram	7	47	52	67,715 + 190,996	238,465 (92.17%)
analytics writer & reader	2	9	4	67,715 + 10,252	74,552 (95.62%)
whatsapp	3	7	7	67,715 + 7,834	74,408 (98.49%)
geo server	1	3	3	67,715 + 2,691	70,300 (99.85%)
calculate sum	1	1	0	67,715 + 544	68,258 (99.99%)

roadmap

time	effect
5 min	first service (vos_calculate_sum) generated and running
20 min	second service (vos_geo_server) generated and running
5-7 days	finished all templates: command of all utilities, multi-service backends
1-2 weeks	full command of modifying/extending the underlying backend
thereafter	best-in-class-level in implementing top class backend services, finishing a year's worth of work every month

in a nutshell

system requirements

• unix operating system (e.g.: mac os, ubuntu, etc ...)
• java 8
• python 2.7.10 (only for using cassandra)
  • to check python version type in the following command in a terminal session python2.7 -V and the output should be Python 2.7.10
  • installing python on mac os
    • download python for mac os
    • open python-2.7.10-macosx10.6.pkg and follow the installer steps
  • installing python on ubuntu
    • cd ~/Downloads
    • wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
    • tar -zxvf Python-2.7.10.tgz
    • sudo apt-get install build-essential checkinstall
    • sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libgdbm-dev libc6-dev libbz2-dev libsqlite3-dev tk-dev libssl-dev
    • cd ~/Downloads/Python-2.7.10/
    • sudo ./configure
    • sudo make altinstall
    • python2.7 --version

prerequisites

• basic java
• basic cassandra (optional for using cassandra database)

quick start example: calculate sum

on YouTube

5-10 min: this tutorial explains how to generate and use the first vangav backend service

vos_calculate_sum is a service that takes a two floats (a and b) request and returns a double (c) response representing the summation of a and b

init

1. create a workspace directory my_services - this is the directory to contain both of vos_backend and all the services generated using it
2. download this vos_backend.zip project (from the green clone or download button up there) inside the workspace directory created in (1) and unzip it
3. rename downloaded vos_backend-master to vos_backend

generate a new service

1. create a new directory my_services/vos_calculate_sum
2. copy controllers.json from vos_backend/vangav_backend_templates/vos_calculate_sum/ to the directory my_services/vos_calculate_sum created in (1)
3. open a terminal session and cd to my_services/vos_backend/tools_bin
4. execute the command java -jar backend_generator.jar new vos_calculate_sum to generate the service
5. enter y for using the config directory in order to use controllers.json for generating
6. enter n for generating a worker service (using workers is explained in a separate section)

writing the service's logic code

• optionally for eclipse users: open eclipse and import vos_calculate_sum project
  • file > import > general > existing projects into workspace > next > set "select root directory" to my_services > under projects make sure that vos_calculate_sum is selected > finish
  • double check the java version used for compiling the project: right click the project > properties > java compiler > enable project specific settings > compiler compliance level > 1.7 or 1.8
• open class HandlerCalculateSum.java under package com.vangav.vos_calculate_sum.controllers.calculate_sum, method processRequest should be as follows in order to complete the request-to-response logic

  @Override
  protected void processRequest (final Request request) throws Exception {

    // use the following request Object to process the request and set
    //   the response to be returned
    RequestCalculateSum requestCalculateSum =
      (RequestCalculateSum)request.getRequestJsonBody();
    
    // set response's value
    ((ResponseCalculateSum)request.getResponseBody() ).set(
      requestCalculateSum.a + requestCalculateSum.b);
  }

start the service

1. cd to my_services/vos_calculate_sum
2. execute the command ./_run.sh

try it out

1. open an internet browser page and type http://localhost:9000/calculate_sum?a=1.2&b=2.3 - this returns 3.5
2. play with a and b values in the request string in (1)
3. try issuing an invalid request (e.g.: set a to "xyz", don't set b, ...) to get a sense of how the default error response looks like (error responses are explained in depth in a separate section)

stop the service

in the terminal session where you started the service press control + d

vangav backend tutorials

on YouTube

1. expand calculate sum to calculator

• shows how to add controllers (api entry points) before and after service generation | on YouTube

2. project's contents

• explains the building blocks of vangav backend | on YouTube

3. next example with database (geo server)

• generates a service that has a backend database and also uses the geo services utility | on YouTube

4. generated service structure

• explains the building blocks of a generated service | on YouTube

5. service generator config structure

• explains the building blocks of the config used to generate a service | on YouTube

6. request object

• when implementing a service's logic, the passed request object keeps all the request's information from start to finish; this tutorial shows how to use that object | on YouTube

7. error response

• explains what happens whenever an error happens during request processing and how to manually return various types of error response | on YouTube

8. using dispatcher - worker(s)

• shows why and how to us the dispatcher with one or more worker service(s) | on YouTube

9. debugging

• explains how to start and use the debugger

10. deploy on a production server

• once your service is ready for release, this tutorial has the step-by-step process till your service is up and running on a production server - as well as how to scale it up | on YouTube

11. vangav backend error codes

• vangav backend detects various types of errors (e.g.: invalid request param, invalid generator config, wrong utility method arguments, unauthorized third-party authentication, invalid vangav mighty solution, ...); this tutorial lists all of vangav backend's error codes/sub-codes with reference to the code producing them for ease of tracing if you get one | on YouTube

utilities, design and src code tutorials

on YouTube

java clients

• handles generating java clients and contains vangav backend's client framework

backend generator

• manages all vangav backend generators (api, database client, worker and java client)

cassandra

• handles generating database clients, contains the framework for generated clients and handles all cassandra operations

compression

• contains different compression algorithms

content: verifiction/formatting/generation

• handles content (code, phriction-wiki, text, ...) checking, formatting and generation; vangav backend relies on this package to verify generation config and format generated code, scripts, wiki, ...

data structures and algorithms

• has data structures (e.g.: heap, kd-tree, tuples, ...) and algorithms (collections, matricies, strings, arrays, ...)

dispatcher - worker

• handles generating worker services, contains the framework for dispatchers/workers and handles all of their operations

exceptions

• vangav exceptions are used to handle bad request and internal error; those exceptions can be returned to the client and loggable (in database, text files, ...)

files

• simplifies various types of files needed for vangav backend services like: properties, json config, images, http response files, ommiting comments, directory operations, ...

geo services

• handles reverse geo coding, geo grids and geo hashing

ids

• handles uuid operations, sequential ids and twitter's snow flake ids

math

• handles mathematical and geometric operations like: numeric, ranges, circles, line segments, straight lines, ...

metrics

• handles distance, time, date and calendar operations

networks

• has sync/async rest client, rest jobs, email clients, twilio messaging and download utility

play framework

• handles generating api code and contains the framework for generated vangav backend services; vangav backend services' backbone

properties

• handles loading and extracting data from properties files

public apis

• handles fetching data from facebook graph api and car2go api

push notifications

• handles building and sending apple and android notifications

security

• handles authentication (facebook, google, oauth 2 and transaction tokens) and cryptography (asymmetric, hashing and two-way encryption)

system

• handles fetching system info (cpu usage, free ram/disk, os type, number of cores, ...) and console operation (interactive, commands, ...)

thread pool

• contains latch threads, periodic jobs and the thread pools responsible for maintaining top performance for vangav backend services (in-memory threads, cassandra, dispatcher and rest client)

vangav mighty

• handles generating and binding vangav mighty solutions | on YouTube

templates

calculate sum

• simple service | on YouTube | but my phone was under 10%

geo server

• a more advanced service with a databse and uses geo services utility | on YouTube | I tried to make it slow!

whatsapp

• service oriented architecture (main + worker + analytics), multi-keyspace database and basic authentication | on YouTube | am I a hologram?

vangav analytics

• service oriented architecture (writer + reader) and generic service design (handles any type of analytics) | on YouTube | it may be foolish to turn this down!

instagram

• on YouTube | [in-spuh-rey-shuh n]

covered topics

• service oriented architecture (main + dispense + jobs + worker + dash board)
• multi-keyspace database
• oauth 2 and facebook authentication
• facebook graph api
• rest jobs
• periodic jobs
• vangav mighty
• push notifications
• logging
• analytics
• client generator
• test and bots services

share

free consulting