jamarparris/flightplan

Search for award inventory using Node.js and Headless Chrome!

Flightplan

Flightplan is a Javascript library that makes it easy to scrape and parse airline websites for award inventory. It uses Puppeteer for scraping, which is built on top of headless Chrome, meaning it behaves just like a full-fledged Chrome browser, but it can be run from the command line with no visible window (allowing you to use your computer for other things). Furthermore, it can run on any platform supported by headless Chrome, which is just about everything (Windows, Mac, or Linux).

Why?

If you're sitting on a pile of airline miles or credit card points, you know that redeeming them can be difficult. Often, planning for my own trips, I would spend hours clicking through an airline's website, searching for available awards, while writing down what I found in a notebook. Eventually, I decided to automate that process, so I could free up my time. Flightplan doesn't scrape much faster than a human would, it simply will do it for hours on end without complaining or making mistakes. This can make planning complex award itineraries much less stressful!

Disclaimer: Scraping is generally against an airline's website's terms of service. As mentioned above, Flightplan typically doesn't place more load on a website than a normal human would, but unlike the human, it can run 24/7. So please use responsibly! Use of any scraping tool (or even excessive non-automated usage) can cause an airline to temporarily (or permanently) ban your IP or member account.

Supported Airlines

Airline	Website	Search	Parse
CX (Cathay Pacific)	AsiaMiles	✅	✅
KE (Korean Air)	SKYPASS	✅	✅
NH (All Nippon Airways)	ANA Mileage Club	✅	✅
SQ (Singapore Airlines)	KrisFlyer	✅	✅

Notes: CX searches are being blocked aggressively, needs more investigation as to the cause.

Geting Started

To use Flightplan, there are a few prerequisites that must be installed:

1. Node.js (Installation instructions: Windows | Mac | Linux)
2. Yarn (Installation instructions)

If you simply wish to install and run the Flightplan tools, run:

yarn global add flightplan-tool

# Create a directory to store data in
mkdir flightplan && cd flightplan

# Explain all available commands
flightplan --help

Before running the search tool, you will need to create a config/accounts.json file. Use the template below:

{
  "CX": [{"username": "XXXXXXXXXX", "password": "password"}],
  "KE": [{"username": "XXXXXXXXXXXX", "password": "password"}],
  "NH": [{"username": "XXXXXXXXXX", "password": "password"}],
  "SQ": [{"username": "XXXXXXXXXX", "password": "123456"}]
}

To add Flightplan to an existing Javascript project, simply use:

yarn add flightplan-tool
# or "npm i flightplan-tool"

Note: When you install Flightplan, it will be bundled with a recent version of Chromium automatically (so you do not need Chrome installed on your machine to use Flightplan).

Running the tools

If you'd simply like to run Flightplan as a stand-alone tool, there are a set of scripts included in the bin directory.

Library usage

With Flightplan, you specify an airline and get back an engine, which supports two operations: searching and parsing.

1. Searching simply takes a query, and fetches the HTML response (or multiple responses for some websites, which break the results up across multiple tabs).
2. Parsing takes those HTML responses, and returns the list of flight awards. (Note: currently, Flightplan ignores non-direct routes and partner awards, this may change in the future.)

This is useful, because searching is expensive, but parsing is cheap. So it makes sense to search once, but be able to parse many times (perhaps due to bug fixes or new features being added).

const fp = require('flightplan');

const cx = fp.new('cx');

(async () => {
  // Must call initialize before searching
  await cx.initialize({ username: '1234567890', 'password': 'passw0rd' });

  // Do a one-way search (replace credentials with real ones below)
  const { responses, error } = await cx.search({
    fromCity: 'HKG', toCity: 'LHR',
    departDate: '2019-03-06', cabin: 'first'
  });
  
  // Check for an error
  if (error) {
    console.log(error);
    return;
  }
  
  // Parse out awards from the responses
  const { awards } = cx.parse(responses);
  console.log(awards);
})();

You can also instruct the search engine to save both the HTML output, and even screenshots! 🎉 This makes debugging what might've gone wrong later much easier. Let's try it out:

const fp = require('flightplan');

const sq = fp.new('sq');

(async () => {
  await sq.initialize({ username: '1234567890', password: '123456' });
  const { htmlFiles, screenshots, fileCount, error } = await sq.search({
    fromCity: 'SIN', toCity: 'HKG',
    departDate: '2019-03-06', cabin: 'business',      
    htmlFile: 'output.html', screenshot: 'output.jpg'
  });
    
  if (!error) {
    console.log('Files Saved:', fileCount);
    console.log('HTML:', htmlFiles);
    console.log('Screenshots:', screenshots);
  }
})();

More API details to come later...