Web Scraper with AWS Lambda and Serverless Framework
devrother / scraping Goto Github PK
View Code? Open in Web Editor NEWwanted site scraping
wanted site scraping
DynamoDB 코드도 추가되면 복잡해질것 같음.
AWS 리소스들을 조작하는 패키지를 따로 만들어 관리.
scrap 한 job data 및 company data를 DynamoDB 에 저장.
asyncio 를 이용하여 효율적으로 scraping.
Test Code 작성
scrap 한 job_id 와 company_id 를 DynamoDB 에 저장하여 관리하여 주기적으로 scrap 할 때 중복해서 scrap 하는 것을 방지할 계획.
parse_data() 메소드의 data 변수에 STRINGS_TO_PARSE_DATA element 가 없으면 KeyError 발생함.
scraping lambda가 image url을 amazon sqs에 전달
aws lambda를 이용하여 scraping.
CI/CD 설정
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.