ribohe94/jsonparsingchallenge

Data Engineering challenge

jsonParsingChallenge

Data Engineering challenge

Input/Output files

Input:
data/clicks/*.json
data/impressions/*.json

Output:
data/output/metrics.json
data/output/recommendations.json
data/output/recommendations.json/_SUCCESS
data/output/recommendations.json/part-00000-b70f97a5-94b4-4dde-8460-8b55ab6ab8a9-c000.json

Steps I took

1. Scala has a deprecated JSON library so I experimented with a few of the most popular out there. I just looked for two things in this library, for it to be lightweight and for it to let me parse a JSON file to a Scala object, and viceversa easily. After some experimentation I opted to use jackson
2. Created JSONETL to handle the reading and parsing of the JSON files.
   1. readJSON(inputDirectory: String)
      1. returns (List[String], List[String]), a list of the click and impression json files respectively.
      2. inputDirectory should be the folder containing clicks/ and impressions/. Using the above folder structure as example, inputDirectory should be "data/". Note: This could be improved by receiving inputDirectory: File instead and avoiding errors in the String path.
   2. parseJSONintoList(stringJSON: String)
      1. Parses a JSON file in string format into a scala object, in this case List[Map[String, Object]]
   3. convertToJSON(result: Any)
      1. Converts Any into Jackson's ArrayNode. Note: It would have been better to specify the type (Iterable[Map[String, Object]]) to ensure Type Safety.
   4. write(result: ArrayNode, path: String = "data/output/metrics.json")
      1. Takes a Jackson's ArrayNode and writes it in json format.
3. Created ClicksImpressionsTransformer to handle the metric calculations
   1. calculateMetrics Receives the list of clicks and impressions and calculates the corresponding metrics.
      1. convertClicksAndImpToScala converts the lists of clicks and impressions to a List per element.
      2. reduceAndRenameClicksAndImp reduces Lists into a single list
      3. groupAndJoinClicksAndImpressions joins clicks and impressions and claculates metrics
4. Trait Mapper contains an ObjectMapper object used to map scala objects into JSON and viceversa
5. SparkInstance contains the SparkSession object used for obejctive #3.
   1. generateRecommendations generates the recommendadions based on the metrics from the previous objective.
   2. write writes DataFrame in spark's json format.