chswx/ldm-ingest

Scripts to process, store, and distribute National Weather Service products in standard formats.

Legacy chswx.com

This is an archive of the source code to the old chswx.com codebase, which has since been folded into a WordPress-based implementation on chswx/wp-theme.

This repository is being archived for posterity, but no new development will be happening here.

ldm-ingest's Issues

Reimplement error logging

Need some basic logging in the beta version beyond just writing to stderr. LDM loses track of that, apparently.

Support storm motion vector (TIME...MOT...LOC) string

Often times special weather statements and warnings will include a TIME...MOT...LOC string from which the time of the storm fix, the storm motion, and the coordinates of the storm can be derived. Add awareness to the parser to send this data along.

Support HAIL...WIND...TORNADO tags

Impact-based warnings come with some machine-readable HAIL...WIND...TORNADO... tags. Ensure these are properly parsed out.

Administrative interface

The bot needs an administrative interface of some kind for easily manipulating configuration.

issued_time is null in all products

When looking at a parsed product in the DB, the issued_time is always null. Need to nail this down.

To fix in 2.1.

pqact needs to change to send WMO, issuing office, date string, corrections, and AFOS as arguments to the ingestor

The incoming arguments should look like this:

--wmo=WMON00 --office=OOOO --time=DDDDDD --afos=AFOS [--corrections=CCA]

This will help generate events directly to parsers without having to figure everything out. Saves some time.

Change when expire date is explicitly displayed in tweets

Expire date should always be shown for tweets indicating an expiry on the following day. Current behavior counts 24 hours ahead and is susceptible to confusion as well as errors when using Daylight Saving Time.

Tropical weather-specific parsers

• Tropical cyclone tracks
• NHC discussions
• NHC public advisories
• HLS products from coastal WFOs

VTEC outlines are already available for tropical weather and will be handled in the interim by the generic parser.

Implement SPC outlook points + text parsing

Parse and store SPC outlook points to allow us to generate maps and alerts based on outlook.

Impact-based warnings

Implement detection of impact-based warnings and expose that data in events so listeners can take advantage of the additional metadata.

Minimum viable API

Begin work on a small API to allow external sites (such as chswx.com) to display our parsed warnings.

Long-running pipe processes

It's inefficient to open and close pipes constantly. Make it to we can have long-running processes to ingest data. Also, make it to where we can explicitly select the parser that will be used to save a few milliseconds of autoloading because I like complexity.

Refactor common text parsers into a standalone library

For reusability's sake, we'll want the parsing tools created in this milestone to be more modular than stuffing everything in a Parser namespace.

Implement pub/sub pattern

This engine needs to be refactored into a publish/subscribe system, which will allow easy implementation of multiple ways to route products (to the Web, to a XMPP chat room, to e-mail, to multiple Twitter accounts, etc.)

Devise a scheme to allow endpoints to subscribe to events generated by the parser. The NWSChat model is not a bad one to follow.

Todos for this to get us back to some sort of operational parity with the current release version:

• Debugger for output in debug mode
• New logging endpoint for overall logging ops
• New Twitter endpoint
• New Email endpoint

Parser refactor for unit testability

Parsing could stand to have some concerns separated for easier unit testability.

Use PHP 7 as the minimum required version

Go big or go home. Performance and language improvements combined with having total control over the servers make this worth it.

Support H-VTEC

At some point, add support for Hydrologic Valid Time Extent Code for better flooding-related metadata.

Implement SPS parsing

Special Weather Statements and Significant Weather Advisories. NWS Charleston, SC does not issue significant weather advisories by name, but many other offices do, so we'll work a little support in.

SPSes can be polygonal or zone-based depending on how they are issued. We should interpret polygonal SPS products in the “significant weather advisory” column. You can also see SPS products for full forecast zones.

Recently, impact-based bullets have made it to the SPS/SWA, so we can perhaps parse those out, too.

Implement job queuing

As the number of events the system processes increases, we'll almost certainly need some sort of queueing backend to handle the load.

Implement VTEC-specific parser

Support polygon warnings

Give parsers the ability to search for and properly encode polygon warnings.

Support piping in of products from the command line

May save on disk I/O and other potentially fun race conditions.

Support multiple VTEC strings

Currently, the ingestor only respects the first VTEC string in a product. However, multiple VTEC strings are often used in the tropical breakpoint products (and can be used in some situations upgrading watches, etc.)

The parser and associated methods need to be updated to permit multiple VTEC strings.

Database schema

Set up tables for the following:

• Convective Watches
• VTEC'd Watches/Warnings/Advisories
• Special Weather Statements
• Mesoscale Convective/Precipitation Discussions
• SPC Outlooks
• Local Storm Reports
• Tropical Products
• River Flood Products
• Forecast data

Implement LSR parsing

Implement parsing local storm reports.

LSRs should export the type, magnitude, description, time, and location (in GeoJSON coordinates) of the report.

Implement mesoscale convective/precipitation discussions (SWOMCD|FFGMPD)

Parsing the mesoscale discussions from SPC allows us to alert on watch probabilities. These may also be helpful for syndication to other advanced platforms.

Parsing mesoscale precipitation discussions may be less helpful for public consumption. Could be good for alerting for more advanced users.

Implement climate product parsing

Allow climate products to be parsed and stored in the database.

Implement Watch Probabilities parsing

We would want to parse watch probabilities so that we can generate graphics with these probabilities, as well as identify watches as PDS watches.

Backing database

Add a backing store for alerts. Backing store is ideally pubsub-capable.

Initial implementation will be RethinkDB, but this should be abstracted away to allow different engines in a future iteration.

• Implement storage initialization for first-time use
• Consider table schema for the DB
• Implement schema for DB
• Set up product routing/saving accordingly