occidere/joballocator

스케줄마다 job 을 할당 및 호출하는 배치

Who am I?

Build data pipeline of products from feeding to processing for expose to search engine in NAVER shopping platform.

• Experienced in build data pipeline of feeding massive data, processing and delivering.
• Skilled in building and operating Elastic Stack for massive indexing, real time searching and visualization.
• Familiar with Hadoop Ecosystem based ETL processing.
• Adepted in running and deploying applications on container system.

Resume

• English: Sungjun_Lee_resume.pdf
• Korean: 이력서.pdf

Don't forget my posts, presentations and contributions too!

• Build monitioring system using Elastic Stack + Lambda
  • Written on D2 (Naver tech blog)
• Configure Kubernetes pods autoscaling by detecting Kafka comsumer lag
  • Presented in NAVER Engineering Day 2019
• 2021 Elastic Bronze Contributor
• 2022 Elastic Gold Contributor
• 2023 Elastic Silver Contributor

Check my additional info here!

Leetcode solving status

Reminiscence

open

My team at Shopping DI in NAVER Corp.

ⓒ 2021. minSW all rights reserved.

joballocator's Issues

Schedule

use schedule;

db.jobinfo.drop();

db.jobinfo.createIndex({"time": 1})
db.jobinfo.createIndex({"status": 1})
db.jobinfo.createIndex({"type": 1, "name": 1}, {unique: true});

db.jobinfo.save({"type": "instagramdump", "name": "twicetagram", "time": "030000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/twicetagram", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "chengxiao_0715", "time": "031000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/chengxiao_0715", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "e_lludda", "time": "032000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/e_ludda", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "wm_ohmygirl", "time": "033000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "redvelvet.smtown", "time": "034000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/redvelvet.smtown", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "jypetwice_japan", "time": "035000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/jypetwice_japan", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "official_izone", "time": "040000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/official_izone", "status": "on"});

//db.jobinfo.update({type: "test", name: "testJob"}, {$set: {time: "002700", jobCallUrl: "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl/"}})

//db.jobinfo.save({"type": "test", "name": "testJob", "time": "003000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl/", "status": "on"});

docker 컨테이너 화

log level 변경

log level 변경

상황

현재 debug 값이 디폴트라서 너무 로그가 많이 남음
이를 info 로 변경할 것

job 호출 시 제대로 호출되지 않는 현상 수정

job 호출 시 제대로 호출되지 않는 현상 수정

상황

• WebClient 로 return 을 기다리지 않고 호출만 하고 끊었더니 제대로 결과가 나오지 않음
  (InstagrampDump 결과 저장이 안됨)

확인

• RestTemplate 나 WebClient 에서 .mono 등으로 blocking 처리 후 정상동작 확인해볼 것

Job 할당 체크 시간범위 JobParameter 로 전환

Job 할당 체크 시간범위 JobParameter 로 전환

상황

현재는 10분이라는 잡 체크 시간 범위가 하드코딩 되어있음
추후 JobParameter 로 변경하여 유동적으로 운영할 수 있도록 개선 필요

Test script

Test Script

command

use schedule;

db.jobinfo.drop();

db.jobinfo.createIndex({"time": 1})
db.jobinfo.createIndex({"status": 1})
db.jobinfo.createIndex({"type": 1, "name": 1}, {unique: true});

db.jobinfo.save({"type": "instagramdump", "name": "twicetagram", "time": "030000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/twicestagram/", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "chengxiao_0715", "time": "031000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/chengxiao_0715/", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "e_lludda", "time": "032000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/chengxiao_0715/e_ludda", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "wm_ohmygirl", "time": "033000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl/", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "redvelvet.smtown", "time": "034000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/chengxiao_0715/redvelvet.smtown", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "jypetwice_japan", "time": "035000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/jypetwice_japan/", "status": "on"});
db.jobinfo.save({"type": "instagramdump", "name": "official_izone", "time": "040000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/official_izone", "status": "on"});

//db.jobinfo.update({type: "test", name: "testJob"}, {$set: {time: "002700", jobCallUrl: "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl/"}})

db.jobinfo.save({"type": "test", "name": "testJob", "time": "003000", "jobCallUrl": "http://localhost:8080/launchJob?url=https://www.instagram.com/wm_ohmygirl/", "status": "on"});