Python bot for crawling data from TMS OpenAPI of CleanSYS automatically. You can set the scheduled time the bot will be started and, also change the time scheduled for bot.
- You can check example in /data.
| mesure_dt | area_nm | fact_manage_nm | stack_code | nh3_exhst_perm_stdr_value | nh3_mesure_value | ... |
|---|---|---|---|---|---|---|
| 2023-01-16 15:00 | 서울특별시 | 노원자원회수시설 | 1 | NaN | NaN | ... |
| 2023-01-16 15:00 | 서울특별시 | 노원자원회수시설 | 2 | NaN | NaN | ... |
- Download this code in your directory.
- Write the auto starting script matched by your environment. Check autostart.md
- Run the script in background on your system.
- Check the data collected by script.
git clone [email protected]:alienatiz/TMS-AutoCrawler.git -b release
Libraries MUST be needed for crawling data by OpenAPI are explained in requirements.txt.
Check them here. > requirements.txt
- The crawled data contains Korean. To save this data as csv data, the encoding format must be set to 'utf-8-sig'.
- OpenAPI server somtimes may be unstable.
- Handling exception is essential depending on errors that may occur in OpenAPI.
- You must get an API Key to use OpenAPI in the Open Data Portal.
- KR 공공데이터포털
- EN Open Data Portal
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent