[FEAT] [High Priority] Meteo Alarm Weather Warnings for VOST Europe

[JorgeMiguelGomes]

[FEAT] [High Priority] Meteo Alarm Weather Warnings for VOST Europe

[miguelantoniosantos]

After a direct chat with @miguelcarreiro, we decided @miguelcarreiro will be making the twitter integration, I (@miguelantoniosantos) will be making the scrapper.

The scrapper will be an module with the following format:

class MeteoAlarm {
   async fetch() {
   }
   async fetchSingle(countryShortcode) {
   }
}
module.exports = new MeteoAlarm();

They async function fetch will return an object with the keys being a country short code and the value being an array.
This array will have multiple (0+) objects, representing an alert, with the following format:

{
      type: '',
      status: '',
      start: '',
      end: '',
      severity: '',
      region: '',
}

I predict having the scrapper done at the end of Friday (02/09/2020 23:59)

[miguelantoniosantos]

The way I found for parsing is:

We get all the countries with warnings with:

let warnings = document.querySelectorAll('.warn.awt');

let country_url = warning[0].parentElement.parentElement.href;

Go through each one that is unique.

Then we have two options, processing the tooltip or opening the region warning and parsing the region page.
I suggest going to the region page as it seems more consistant.

let warnings = document.querySelectorAll('.warn2.awt2');

let text = warnings[0].parentElement.parentElement.getAttribute('onclick');

let regex = /transfer_to_area\('([^']+)'\)/m;

let region_url= text.match(regex)[1]);

And in the region page we parse the region name warnings and all the data needed.

@miguelcarreiro seems good?

[miguelcarreiro]

The way I found for parsing is:

We get all the countries with warnings with:

let warnings = document.querySelectorAll('.warn.awt');

let country_url = warning[0].parentElement.parentElement.href;

Go through each one that is unique.

Then we have two options, processing the tooltip or opening the region warning and parsing the region page.
I suggest going to the region page as it seems more consistant.

let warnings = document.querySelectorAll('.warn2.awt2');

let text = warnings[0].parentElement.parentElement.getAttribute('onclick');

let regex = /transfer_to_area\('([^']+)'\)/m;

let region_url= text.match(regex)[1]);

And in the region page we parse the region name warnings and all the data needed.

@miguelcarreiro seems good?

@miguelantoniosantos Looks good, although Meteoalarm also has a RSS feed with the list of countries, and then each country have it's own RSS feed with all regions.
I watched RSS feeds, and it has some problems, for example, the only way to see if a warning was issued is by looking at the image URL, whereas in source code we can check 'warn2' class attribute), so I think that scraping would be a better solution.

[JorgeMiguelGomes]

@miguelantoniosantos & @miguelcarreiro how long will it take to implement this? Real expectations, please

[JorgeMiguelGomes]

I predict having the scrapper done at the end of Friday (02/09/2020 23:59)

Sounds great!

[miguelantoniosantos]

Created a gist with an example output of MeteoAlarm.fetch()

Here is an example use of the scrapper:

const MeteoAlarm = require('./api/MeteoAlarm');


async function get() {
  const portuguese_data = await MeteoAlarm.fetchSingle('PT');
  const all_data = await MeteoAlarm.fetch();
}

get();