jacob119/utility_scala

Scala programming language basic & functional programming (fp) & Spark Scala basic demo | #SE

Utility_Scala

Introduction of Scala programming language as well as Spark Scala via couples of basic scripts in common use cases. Please check the Main files for more information.

• Scala cheatsheets
• Scala learn material
• Scala/Spark Ref

Scala Projects

• Scala Coursera - My note/code for Coursera Scala series class
• spark_emr_dev - Demo of submitting Hadoop ecosystem jobs to AWS EMR
• spark-etl-pipeline - Demo of various Spark ETL processes
• utility_Scala - Scala/Spark programming basic demo

Note

• Scala Note - Scala learning note

Main Files

1. Utility Scala - Scala scripts for utility

2. Spark Scala Demo - Scala spark basic demo

3. Scala basic - variable, data structure

   • Scala_basic_demo_3 - Scala basic data structrue : array, list, tuple, apply function, lambda with them basic. And map, exception demo
   • Array
     • ArrScalaArrayDemoay
     • ArrayDemo1
     • Scala Array basic op, properties
   • List
     • ListDemo_1
     • Scala_yield_to_List
     • Scala Lists are quite similar to arrays, which means all the elements of a list have the same type - but there are two important differences.
     • First, lists are immutable, which means elements of a list cannot be changed by assignment.
     • Second, lists represent a linked list whereas arrays are flat. The type of a list that has elements of type T is written as List[T].
   • HashMap
     • HashMap
     • Scala Hash Map basic op, properties
     • MapDemo_1
     • Scala Map basic op, properties
   • Enumeration - Scala Enumeration basic op, properties
   • Nothing_demo1, Nothing_demo2 - Nothing can still pass information when the program failed. null can be passed to AnyRef, but NOT AnyValue (AnyValue : Int, Float, Char, Boolean, Long, Double, Byte, short...)
   • Either, Left, Right
     • EitherLeftRight1
     • EitherLeftRight2
     • EitherLeftRight3
     • EitherLeftRight4
   • Loop
     • ForLoop1
     • ForLoop2
     • ForLoop3
     • ForLoop4

4. Scala basic - function, class, case class, constructor...

   • Class - Scala class basic op, properties
   • Case Class, Case Class2 - Scala class basic op, properties
   • ClassPolymorphismDemo1 - Scala class Polymorphism Demo
   • ScalaDefaultValueDemo1 - Scala Scala Default Value
   • ScalaIfElseMatchForloop - Scala if, else, while, match basic
   • ScalaFunction_1 - Scala function, lambda basic
   • ScalaFunction_2 - Scala function with dynamic var length
   • Scala_basic_demo_4 - Scala try, exception, error handling demo
   • ScalaFileIODemo, ScalaFileIODemo2, ScalaFileIODemo3 - Scala file IO basic
   • ScalaLazyDemo - Scala lazy value demo
   • ScalaUnitDemo - Scala function with no return value called as procedure, with return type Unit
   • Override
     • OverrideDemo
     • OverrideDemo2
     • OverrideDemo3
     • OverrideDemo4
     • OverrideDemo5
     • Scala Override basic op, properties
   • Higher Order function
     • HigherOrderfuncDemo
     • HigherOrderFuncDemo_2
     • HigherOrderFuncDemo_3
     • HigherOrderFuncDemo_4
     • HigherOrderFuncDemo_5
     • Higher Order Functions are functions that take functions as arguments and/or return functions.
   • RecursionDemo - Scala Recursion basic op, properties
   • Constructor
     • Constructor can define value to its attribution when create the object
     • can has as many as posssible constructor methods in scala
     • "main constructor", "support constructor"
     • scala constructor format

      class ClassName{parameters}{ // main constructor
      	// code
      	//
      	def this(parameters){//support constructor
      	}
      	def this(parameters){// can have multiple support constructors ...
      	}
      }

     • java ref
     • Constructor
       • ConstructorDemo1
       • ConstructorDemo2
       • ConstructorDemo3
       • ConstructorDemo4
       • ConstructorDemo5
   • PatternmatchDemo, PatternmatchDemo2, PatternmatchDemo3, PatternmatchValueCompare - Scala Pattern match basic op, properties
   • thisDemo, - thisDemo2 - Scala this basic op, properties
   • SomeDemo - Scala Some basic op, properties
   • CaseHeadTailNil - Scala Case on List Head Tail Nil basic op, properties
   • GenericTypeParam - Scala Generic Type basic op, properties, e.g. : def exec[T](f: () => T): T = {f()}
   • TryGetOrElse, TryGetOrElse2 - Scala try GetOrElse(else) example
   • UpperCass - Scala UpperCass ( <: ) basic op, properties
   • Find - Scala Find, exists, contains, and isDefined examples
   • Partial applied functions demo 1 - Scala Partial applied functions examples
   • Partial functions demo 1 - Scala Partial functions examples
   • Bean Property - @BeanProperty example
   • TypeConvert1, TypeConvert2- Scala type convert : upper transform, lower transform : asInstanceOf, check if object's class type : classOf, getClass.getName
   • example

    // *** declare a Employer_03 instance, but transform it to Person_03 class
    val emp:Person_03 = new Employer_03
    // *** transform emp to Employer_03 class (asInstanceOf)
    emp.asInstanceOf[Employer_03].sayHello() // downward transform

   • SuperConstruct, SuperConstruct1 java, SuperConstruct2, SuperConstruct3- Scala/Java Super constructor examples

5. Scala class, trait, abstract class..

   • Class
     • Method create class instance in Scala
       • New className
       • Apply : example
       • Dynamic mixing : example
       • Anonymous sub class
   • Trait
     • Trait
     • TraitAbstract
     • TraitDemo2
     • TraitDemo3
     • TraitDemo4
     • TraitDemo5
     • TraitDemo6
     • TraitDemo7
     • TraitDemo8
     • TraitDemo9
     • TraitDemo10
       • Compare the ordering with trait construct ways
         • way 1) create class instance -> class hasn't been created yet when mixing trait
         • way 2) create "anonymous" sub class -> class already been created when mixing trait
     • TraitDemo11
     • TraitDemo12
     • TraitAsInterface
     • TraitAbstractOverwrite1
     • Scala trait : a "supplement" of scala Inheritance (scala only allows "single Inheritance", e.g. a class can only have one parent class), so trait offers more flexibility
     • Scala trait can do sth like "java interface"
     • trait can have both abstract method and regular method (a method that Not implemented in trait is the "abstract" method)
     • ALL Java interface can be used in scala as Scala trait
     • if there are "composition" traits (class className extends trait1 with trait2 with trait3 ...) when creating, then scala will
       • declare the instance from left to right
       • execute the method from right to left
     • Scala TraitDemo, Trait Abstract basic op, properties. NOTICE : Scala traits don’t allow constructor parameters (so that's why we use abstract class)
     • pattern

      // if no parent class
      class className extends trait1 with trait2 with trait3 ...
      //
      // if there is parent class
      class className extends parentclassName with trait1 with trait2 with trait3 ...

     • TraitMixin1
     • TraitMixin2
     • TraitMixin3
     • TraitMixin:
       • Can mixin trait when construct the object (class) => to extend functionality
       • also can implement on abstract class
       • TraitMixin only works in Scala (not Java)
       • Can do the extension but not change inheritance status
   • Abstract Class
     • Abstract Class
     • AbstractDemo
     • Scala also has a concept of an abstract class that is similar to Java’s abstract class. But because traits are so powerful, you rarely need to use an abstract class. In fact, you only need to use an abstract class when:
     • You want to create a base class that requires constructor arguments
     • Your Scala code will be called from Java code
   • Class VS Object 1, Class VS Object 2 - Compare Class, object difference, feature in Scala
   • Type Parameterlization Demo1 - Type Parameterlization in Scala
   • Apply
     • Apply method demo 1
     • Apply method demo 2
     • Apply method demo 3
     • Apply method demo 4
     • Apply method demo 5
     • via apply method in the object, one don't need to use object.apply(sth) for calling the method, but can use object(sth) directly
     • if we have 1 object : Object(param1, param2...) then we will get the Companion class object return via the Apply method
   • anonymousClass_java, anonymousClass- *** Override/implement abstract attribution/method.. in java & scala
   • Nest class
     • NestedClass1
     • NestedClass2
     • NestedClass3
     • is similar as inner class in java.
     • definition : if one completed class is inside the other class => nest class

      // java example 1
      class Outer{  // outer class
      	class inner{  // inner class
      // code
      	}
      }
      class Other{ // outer other class
      // code
      }

      // java example 2
      class OuterClass{  // outer class
      	class InnerClass{ // inner class
      		publice void test (InnerClass ic){
      			System.out.Prlintln(ic);
      		}
      	}
      	static class StaticInnerClass {  // static inner class
      		// code
      	}
      }

6. Scala object

   • LoadPackageDemo1, LoadPackageDemo2, LoadPackageDemo3, LoadPackageDemo4 - Scala load package examples

   • pkgObject - package can have class, object, and trait... but it CAN'T HAVE function, var... In order to solve it, scala offers the package object concept

   • Singleton pattern

     • Only allow some classes be existing in some specific class instances
       • -> pros : can save resources
       • -> cons : Thread safety concern
       • example : Runtime class in java
       • ref Java1
       • SingletonDemo java
     • SingletonDemo1
     • SingletonDemo2

   • Companion

     • CompanionDemo1
     • CompanionDemo2
     • CompanionDemo3
     • CompanionDemo4
     • CompanionDemo5
     • Companion is the combinaton of class static method/value.. + class basic method/value..
     • Scala Companion demo (An object that has the same name as a class is called a companion object of the class, and it is often used to contain factory methods for the class that it complements)
     • Since there is no static class/method.. in Scala, so Scala offers the Companion for similIar functionality
     • In development, we put the basic attribution, method ... in Companion class ; and we put the static consents in the Companion object

   • VisibilityDemo1 - extend package visibility. e.g. : private[ScalaBasic] val name = "jackkkk"

   • ImportPackage1 - Scala import package demo

7. Scala implicit

   • implicit is the way that you dont need to pass parameters explicitly in functions in Scala, but Scala will be able to find them from the implitict scope once you defined them. Use implicit can make your function more general and easy to import/deal with different cases per pattern
   • Implicit_Demo1, Implicit_Demo2, Implicit_Demo3, Implicit_Demo4, Implicit_Demo5, Implicit_Demo6, Implicit_Demo7, Implicit_Demo8, Implicit_Demo9 - Some implict examples
   • Implicit_1
   • Implicit_2
   • Implicit_3
     • More implicit demos from utube tutorial
     • implicit func ONLY ACCEPTS ONE ARGUMENT
     • the implicit func will BE USED AUTOMATICALLY WHEN THERE IS SUCH CASE
     • the NAME of implicit func IS NOT MATTER; INPUT AND OUTPUT DTYPE IS THE (IMPORTANT) THING
     • there can be multiple implicit func, but have to make sure THERE IS ONLY ONE implicit func CAN BE REFERRED IN CURRENT STATE
   • implicit_transformation_demo_1, implicit_transformation_demo_2 - implicit transformation can automatically transform "high accuracy" val to "low accuracy". e.g. : Byte -> Short, Short -> Int, Int -> Long, Long -> Float, Char -> Int ..
   • ImplicitParameters, ImplicitFunc, ImplicitClass - Scala implicit in Parameters, func, class

8. Scala OOP

   • Features : Encapsulation, Inheritance, Polymorphism

     • ref

   • AbstractDemo1

   • encapsulationDemo1, encapsulationDemo2

     • encapsulation is one of the features in OOP, abstract the data and methods and encapsulate them, so only the "defined" method can be implimented to the data
     • Pros on encapsulation
       • hide the implementation details
       • validate the data, make it safe and feat the business needs
     • Steps do encapsulation:
       • step1) "private" the method/attributions
       • step2) offer "public" methods (e.g. : getter, setter or @BeanProperty...) (getter : set attribution value, getter : get attribution value)

   • OOP hw 1

9. Scala Script example

   • DecoratorDesignPattern - Scala Decorator Design Pattern example
   • DirectoryOP - Scala Directory OP example
   • Show run time elapse - show how long the code running in Scala
   • Scala CLI args
     • ScalaGetArg1
     • ScalaGetArg2
     • ScalaGetArg3
     • ScalaGetArg4
     • Scala load arguments from CLI

10. Scala exception, error handling

    • ExceptionDemo1
    • ExceptionDemo2 - format : try - catch - finally
    • • there is NO compile exception in Scala (only java has), all exceptions in Scala happen during runtime. All exceptions inherit from the "throwable" class, which is the class with "nothing" type that is acceptable to all class/method...

11. Others

12. Spark basic

    • Spark_basic_demo_1 - Spark function basic demo
    • Spark_basic_demo_2 - Spark RDD basic op demo : count avg friends by age
      • Spark_basic_demo_2 extra - Extra : count avg friend by first name
    • Spark_basic_demo_3 - Spark RDD filter demo : min temp on each station (weather data)
    • Spark_basic_demo_4 - Spark RDD flatMap demo : word count (book data)
      • Spark_basic_demo_4 extra - Extra : word count with regular expression preprocess
    • PopularMoviesNicer - Spark broadcast demo : movie-lens data (ml-100k)
    • MostPopularSuperhero - Lookup values from RDD and map them to another RDD, and options method in Spark
    • DegreesOfSeparation - Code BFS (Breadth-first search)from scratch via spark
    • MovieSimilarities - Code the Collaborative filtering (CF) (item based) from scratch via Spark
    • SparkSQL - Spark SQL basic
    • DataFrames - Spark Dataset, a important feature when in Spark-2.0/SparkMlib/Spark-stream
    • PopularMoviesDataSets - Do the popular movie computation again via the Spark datasets
      • PopularMovies - Original popular movie approach via Spark RDD
    • GraphX - Spark GraphX demo : compute the social network degree for the marvel heros

Quick Start

Quick start manually

# DEMO 1) run scala hello world 
$ git clone https://github.com/yennanliu/utility_Scala.git
$ cd utility_Scala
$ scala src/main/scala/UtilityScala/HelloWorld.scala 
#$ Hello World


# DEMO 2) run scala spark hello world via sbt 
$ cd utility_Scala
$ sbt package
$ sbt
# inside sbt console
sbt:Simple Project> run 
# [warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list

# Multiple main classes detected, select one to run:

#  [1] AnonymousFuncDemo
#  [2] ClassDemo
#  [3] FileIODemo
#  [4] ForLoopDemo
#  [5] FunctionChangeableParameterDemo
#  [6] FunctionCompositionDemo
#  [7] HelloWorld
#  [8] IfElseDemo
#  [9] OperatorDemo
#  [10] PatterMatchDemo
#  [11] SimpleApp
#  [12] Test
#  [13] UderDefinedDefaultParamFuncDemo
#  [14] UderDefinedFuncDemo

Enter number: 11

# [info] Running SimpleApp 
# ...
#  >>>>>>>>>>>>>> OUTPUT
# Lines with a: 21, Lines with b: 9
#  >>>>>>>>>>>>>> OUTPUT
# ...

# DEMO 3) run scala spark hello world
$ cd utility_Scala
$ sbt clean compile && sbt assembly 
$ spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/scala-2.11/simple-project_2.11-1.0.jar

# REPL via sbt console
$ sbt
console
scala> 

# ✘ yennanliu@MacBook-Pro  ~/utility_Scala   master ●  
# ✘ yennanliu@MacBook-Pro  ~/utility_Scala   master ●  sbt  
# [info] Loading settings for project utility_scala-build from plugins.sbt ...
# [info] Loading project definition from /Users/yennanliu/utility_Scala/project
# [info] Loading settings for project utility_scala from build.sbt ...
# [info] Set current project to UtilityScala (in build file:/Users/yennanliu/utility_Scala/)
# [info] sbt server started at local:///Users/yennanliu/.sbt/1.0/server/ff2f518f2235c5fb0743/sock
# sbt:UtilityScala> console
# [info] Starting scala interpreter...
# Welcome to Scala 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_252).
# Type in expressions for evaluation. Or try :help.

# scala> import slick.driver.H2Driver.api._
# import slick.driver.H2Driver.api._

# scala> 

Quick start via Docker

$ git clone https://github.com/yennanliu/utility_Scala.git
$ cd utility_Scala
$ docker build . -t spark_env
$ docker run  --mount \
type=bind,\
source="$(pwd)"/.,\
target=/utility_Scala \
-i -t spark_env \
/bin/bash

Quick start via `Spark-submit`

# package the scala saprk scripts
$ sbt package
# list the current classes
$ ls target/scala-2.11/classes
# run ForLoopDemo
$ spark-submit \
  --class ForLoopDemo \
  target/scala-2.11/utilityscala_2.11-1.0.jar 
# run LambdaFuncDemo
$ spark-submit \
  --class LambdaFuncDemo \
  target/scala-2.11/utilityscala_2.11-1.0.jar 
# run spark_basic_demo_4
$ spark-submit \
  --class SparkBasic.spark_basic_demo_4 \
  target/scala-2.11/utilityscala_2.11-1.0.jar
# run MovieSimilarities
$ spark-submit \
  --class SparkBasic.MovieSimilarities \
  target/scala-2.11/utilityscala_2.11-1.0.jar 50 

Development

• Trouble shooting

  • if facing Origin location must be absolute error in building a SBT project like errors when sbt build (sbt package), try to clean the sbt cache via rm -fr ~/.ivy2 (remove entire .ivy2 file), then build the project again
  • https://www.scala-sbt.org/1.x/docs/Dependency-Management-Flow.html

• Clean cache (sbt) : in case when there is issue build the project via IntelliJ sbt

ls -al
rm .idea

scala-learn-material

scala-learn-material

• Scala Tutorial

  • https://docs.scala-lang.org/tour/basics.html
  • https://www.handsonscala.com/index.html
  • https://github.com/handsonscala/handsonscala
  • https://www.javatpoint.com/scala-tutorial
  • https://www.tutorialspoint.com/scala/
  • http://www.runoob.com/scala/scala-basic-syntax.html
  • https://ithelp.ithome.com.tw/users/20107343/ironman/1301?page=1

• Tour on Scala

  • https://docs.scala-lang.org/tour/tour-of-scala.html

• Scala Stadard Library API

  • https://www.scala-lang.org/api/current/

• Scala on stackoverflow

  • https://stackoverflow.com/tags/scala/info

• Scala Quick Ref

  • https://courseware.epfl.ch/courses/course-v1:EPFL+progfun2+2018_T1/6c4ec266ee24467c9745009ba131f3da/

• Scala books

  • https://github.com/yennanliu/data_science_repo/tree/master/book/scala

• Scala trait intro

  • https://docs.scala-lang.org/overviews/scala-book/traits-intro.html

Ref

Ref

• Scala Regular Expression

  • https://www.tutorialspoint.com/scala/scala_regular_expressions.htm
  • https://www.geeksforgeeks.org/regular-expressions-in-scala/

• Build Scala Spark project with sbt

  • http://xd-deng.com/render_html/step_by_step_to_package_spark_app_scala.html

• sbt manual

  • https://www.scala-sbt.org/1.x/docs/index.html

• Write test for Scala

  • https://www.scalatest.org/

• scala test

  • http://www.scalatest.org/user_guide/using_scalatest_with_sbt

• Scala spark source code tutorial

  • https://github.com/lw-lin/CoolplaySpark

• Kafka with scala spark demo

  • https://github.com/spirom/spark-streaming-with-kafka

• Spark scala test

  • http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/

• Spark scala tutorial

  • http://allaboutscala.com/big-data/spark/

• Scala DB client API

  • http://scala-slick.org/doc/3.0.0/introduction.html
  • https://www.playframework.com/documentation/2.7.x/ScalaDatabase