fexd/saskwildfire

A bot to propogate wildfire information in Saskatchewan. https://twitter.com/SaskWildfire

👋 Arlin Schaffel (he/him) 🚴💨

I'm a software developer and technology educator; focused on front-end development, mobile applications, and UI/UX design. Over the past decade I've helped launch dozens of websites, shipped hundreds of apps/games, and taught hundreds of students about technology.

• 💼 linkedin.com/in/arlin
• 🐘 hachyderm.io/@FeXd
• 🌐 fexd.com
• 🎓 arlin.education

🛠️ Methodologies, Languages, Tools

I've used a lot of different things over the years. Things in bold I'm actively using.

• 📊 Agile Software Development, Test Driven Development, Servant Leadership, Adult Learning, Clean Code
• 🧮 JavaScript, Python, Swift, Java, Objective C, C#, TypeScript, React, React-Native, HTML, CSS
• 💻 VSCode, Xcode, Android Studio, Git, SourceTree, Adobe CC, Microsoft 365, Google Workspace, Unity, Figma

👨‍🏫 Tips for Students

If you aren't a student, keep scrolling.

1. ask for help and be kind to yourself
2. use dev tools, always read your console, and check your inspector
3. git commit early and often, commit messages longer your code are ok
4. good communication is more important than most things

saskwildfire's Issues

Fires to Date

Apparently this PDF exists: http://environment.gov.sk.ca/firefiles/firestodate.pdf

Found via this page: https://www.saskatchewan.ca/government/notarized-documents-legislation-maps/maps

firestodate.pdf

Could be another map to tweet about.

pdf2image ValueError

Traceback (most recent call last):
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 479, in pdfinfo_from_path
    raise ValueError
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/arlin/Git/saskwildfire/main.py", line 92, in <module>
    generate_images_from_pdf('./pdf/', item['pdf'], './image/')
  File "/Users/arlin/Git/saskwildfire/main.py", line 51, in generate_images_from_pdf
    images = convert_from_path(in_path+pdf)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 98, in convert_from_path
    page_count = pdfinfo_from_path(pdf_path, userpw, poppler_path=poppler_path)["Pages"]
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 488, in pdfinfo_from_path
    raise PDFPageCountError(
pdf2image.exceptions.PDFPageCountError: Unable to get page count.
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't read xref table

NewConnectionError

Traceback (most recent call last):
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x109114370>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='environment.gov.sk.ca', port=80): Max retries exceeded with url: /firefiles/DailyFireDangerMaps/tomorrow_fwi.pdf (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x109114370>: Failed to establish a new connection: [Errno 60] Operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/arlin/Git/saskwildfire/main.py", line 89, in <module>
    download_file(item['path'], './pdf/', item['pdf'])
  File "/Users/arlin/Git/saskwildfire/main.py", line 39, in download_file
    response = requests.get(url+file, stream=True)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='environment.gov.sk.ca', port=80): Max retries exceeded with url: /firefiles/DailyFireDangerMaps/tomorrow_fwi.pdf (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x109114370>: Failed to establish a new connection: [Errno 60] Operation timed out'))

Rainfall Map

After a recent comment on twitter, I started looking into if there was an official rain / precipitation report form the Government of Saskatchewan. The Ministry of Agriculture has two reports: https://www.saskatchewan.ca/business/agriculture-natural-resources-and-industry/agribusiness-farmers-and-ranchers/programs-and-services/information-services-for-agribusiness-farmers-and-ranchers/maps-for-farmers-and-agribusiness

• Weekly Rainfall: https://publications.saskatchewan.ca/#/products/113951
• Cumulative Rainfall: https://publications.saskatchewan.ca/#/products/113952

The PDFs live in publications.saskatchewan.ca and are available via their API: https://publications.saskatchewan.ca/api/v1/products/113952/formats/128248/download

Look into how easy it would be to figure out the URL and regularly download.

Move Server

The mac mini that I run this bot from occasionally has issues. Wouldn't it be nice to run it somewhere else? I'm not super interested in having a second service to do this, but was reminded that GitHub Actions might be a solution?

There is this Twitter Bot Action that is JavaScript only, but I'm sure there would be a similar solution with Python: https://github.com/marketplace/actions/twitter-bot-action

Migrate from Twitter

With twitter 🔥 it is important to have options for other services. Mastodon support would be great, but also just a simple upload (FTP?) to a website would be great. I have purchased https://www.saskwildfire.com/ so it's possible that this may be the time to move over there as the "primary" place to find information?

FileNotFoundError

Crash from last month.

2022-03-31T11:29:39Z: Error: generate_images_from_pdf! Unable to get page count.
Syntax Warning: May not be a PDF file (continuing anyway)
Syntax Error (2): Illegal character <68> in hex string
Syntax Error (3): Illegal character <74> in hex string
Syntax Error (4): Illegal character <6d> in hex string
Syntax Error (5): Illegal character <6c> in hex string
Syntax Error (8): Illegal character <68> in hex string
Syntax Error (14): Illegal character <74> in hex string
Syntax Error (15): Illegal character <69> in hex string
Syntax Error (16): Illegal character <74> in hex string
Syntax Error (17): Illegal character <6c> in hex string
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't read xref table
 
Traceback (most recent call last):
  File "/Users/arlin/Git/saskwildfire/main.py", line 110, in <module>
    if not os.path.isfile('./image_old/'+item['pdf']+'0.png') or not filecmp.cmp('./image/' + item['pdf'] + '0.png', './image_old/' + item['pdf'] + '0.png', shallow=False):
  File "/Users/arlin/.pyenv/versions/3.9.0/lib/python3.9/filecmp.py", line 52, in cmp
    s1 = _sig(os.stat(f1))
FileNotFoundError: [Errno 2] No such file or directory: './image/today_fwi.pdf0.png