x10combo/flight-prices-api

API which is designed to provide users with a way to retrieve flight prices between different airports.

Flight Prices API Documentation

The Flight Prices API is designed to provide users with a way to retrieve flight prices between different airports. This API utilizes web scraping techniques to fetch flight information and prices from the Ryanair website.

Table of Contents

• Getting Started
  • Prerequisites
  • Installation
• Usage
  • Endpoint
  • Request
  • Response
  • cURL
  • Puppeteer Automation
  • Masking
  • Unit Testing

Getting Started

Prerequisites

Before you begin, make sure you have the following tools and dependencies installed:

• npm
• Laravel (Including the Artisan tool)
• Postman
• Composer (for installing PHP libraries)
• Puppeteer

For traffic monitoring and proxy/masking services:

• Charles

Installation

npm

• Clone this repository to your local machine.
• Navigate to the project directory using the terminal.
• Install the required Node.js packages by running the following command:

npm install

Laravel

1. Open a terminal or command prompt.
2. Make sure you have Composer installed. If not, download and install Composer from getcomposer.org.
3. Navigate to the directory where you want to install Laravel.
4. Run the following command to create a new Laravel project:

composer create-project --prefer-dist laravel/laravel flight-prices-api

5. Once the installation is complete, navigate into your project directory:

cd flight-prices-api

7. Copy the .env.example file and rename it to .env:

cp .env.example .env

8. Generate an application key by running the following command:

php artisan key:generate

9. Configure your database settings in the .env file. It should be something like this:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=your_database_name
DB_USERNAME=your_database_user
DB_PASSWORD=your_database_password

10. Run migrations to create the necessary database tables:

php artisan migrate

11. Your Laravel installation is now complete! You can start the development server using:

php artisan serve

Postman

Step 1: Download Postman

Go to the official Postman website:

https://www.postman.com/

Click on the "Download the App" button to start downloading the installer for your operating system.

Step 2: Install Postman

Once the installer is downloaded, follow these steps:

1. Run the installer executable.
2. Follow the on-screen instructions to install Postman.
3. After the installation is complete, launch Postman.

Composer

1. Install PHP:
   Download the Windows PHP binaries from the official PHP website. Make sure to select the version that matches your system architecture (x64 or x86). Extract the downloaded zip file to a directory of your choice (e.g., C:\php).
2. Install Composer:
   Download and install Composer by running the Composer Windows Installer from the official Composer website. Follow the installation instructions in the installer.
3. Verify Installation:
   Open a new Command Prompt (CMD) window and run the following command to verify that Composer is correctly installed:
   composer --version
   This should display the version of Composer you installed.
4. Create composer.json File:
   In your Flight Prices API project directory, create a file named composer.json. This file lists the PHP packages or libraries your project depends on. Here's an example of what the structure should look like:
   

   "require": {
           "php": "^8.0.2",
           "guzzlehttp/guzzle": "^7.7",
           "laravel/framework": "^9.19",
           "laravel/sanctum": "^3.0",
           "laravel/tinker": "^2.7"
       },

5. Install Dependencies:
   Open a Command Prompt (CMD) window and navigate to your project directory. Run the following command to install the dependencies listed in your composer.json file:
   composer install
   Composer will read the composer.json file, download the required packages, and set up autoloading.

Puppeteer

Before you begin, make sure you have the following prerequisites installed on your system:

• Node.js (version 10 or higher)
• npm (Node.js package manager)

1. Install Node.js and npm:
   If Node.js and npm are not already installed, you can download and install them from the official Node.js website.
2. Create a New Project Directory:
   Open a terminal or command prompt and navigate to the directory where you want to create your Puppeteer project. Run the following command to create a new directory and initialize a new Node.js project:
   mkdir flight-prices-api
cd flight-prices-api
npm init -y
3. Install Puppeteer:
   In your project directory, run the following command to install Puppeteer as a dependency:
   npm install puppeteer
4. Write Your First Puppeteer Script:
   Create a new JavaScript file (scrape.js) in your project directory. Inside the file, write a simple Puppeteer script to open a webpage and take a screenshot:
   

   const puppeteer = require('puppeteer');
   (async () => {
   const browser = await puppeteer.launch();
   const page = await browser.newPage();
   await page.goto('https://www.ryanair.com/gb/en/cheap-flights');
   await page.screenshot({ path: 'screenshot.png' });
   await browser.close();
   })();

5. Run Your Puppeteer Script:
   In your terminal, run the following command to execute your Puppeteer script:
   node scrape.js
   Puppeteer will launch a headless browser, open the specified URL, take a screenshot, and save it as screenshot.png.

Usage

API Access

Beforehand, we need to visit https://www.ryanair.com/flights/gb/en.

After we have typed our flight specifics beforehand, the link should look something like this:

Searching anything at all will provide us with a URL which we need to send a GET request to via Postman. It is crucial to find this, which can be done by opening Inspect Element (F12) and accessing the Network tab:

Make sure to refresh the site and open Network to find the mentioned URL. We can now access the flight API of the site.

NOTE: This API URL is related to a specific flight which was available at testing. It may stil be available at the time of writing this, but if it's not, just send another request as the steps mention with different (correct) details and there should be no problems.

Endpoint

The API provides a single endpoint for retrieving flight prices:

• Endpoint: /flights
• HTTP Method: POST

Request

The /flights endpoint expects a JSON body with the following parameters:

• departure: The three-letter code of the departure airport.
• arrival: The three-letter code of the arrival airport.
• departure_date: The date of departure in the format 'YYYY-MM-DD'.

Here's our GET request done in Postman using the URL we mentioned in the Usage tab:

Response

The API responds with a JSON object containing a list of available flights and their corresponding prices, which can be accessed by sending a POST request to our URL which in this case is http://localhost:8000/api/flights. The response format is as follows:

One important thing to note is that you will be shown the cheapest flight for the specific parameters that you enter. For example, below you can see that I have searched for a flight which departs from Tirana (TIA) and lands in Catania (CTA). This specific flight is designated for November, so the value of "departure_date" should be the last of the month. Since the request I sent matches the flight data on the website, I am presented with the specific data as well as the price, which I was looking for.

The reason why it matters for the data entered to be correct is that if we search for a flight with data values that are not present in the website, we will get an empty array, which of course means that flight does not exist. (Notice the intentional typo that I have made here)

cURL variant:

The last step would be to commit a POST request using cURL as an alternative to this process. This will allow us to view the flight data from within the terminal, if the user wants to.

This can be done by inputting this code snppet into a terminal, preferably Bash: (NOTE: Do not close your current Bash terminal because that will terminate your laravel server. Open a new one.)

curl -X POST -H "Content-Type: application/json" -d '{ "flight_number": "ABC123", "origin": "New York", "destination": "Los Angeles", "departure_time": "2023-08-07T12:00:00" }' http://127.0.0.1:8000/api/flights

As a result, you will be returned the raw flight data. You can also choose for the terminal to show you that the data was retrieved successfully in a statement by adding it into the FlightController.php file, but that may or may not provide you with the data itself, so I chose this as an alternative instead. The reply should look like this:

Puppeteer Automation

After installing the required dependencies, the first thing you should do is create the basic puppeteer structure in your .js file. You can name it whatever you want, and then execute it via "node [filename.js]". Everything within (async() will be part of the automated script.

const puppeteer = require('puppeteer');

(async () => { <---scraping logic and proxy functionality---> })();

For example, as you can see below, the automated actions go as follows: The program first waits for a connection to the proxy server, then the readline node module is used to obtain the input data from the user regarding the flight parameters. They are then applied to the URL, which Puppeteer then travels to. Since we have headless set to true, we do not see the browser being opened and performing actions. Rather, we get our results after a short delay which is caused by the proxy service making sure we don't get blacklisted.

The data is then gathered into a readable and compacted format and presented to the user with the values they requested.

Any puppeteer script is executed with the node [....].js command from within your project's root directory in a terminal, after you've made sure the server and proxy service are up and running.

The end result is this easy and accessible user display:

Masking

However, you need to keep in mind that the above result is only possible once you have a stable connection to Charles and have successfully set up your SSL proxying settings. This is how you're supposed to do that:

After executing all the above steps, you should make sure to open Charles and connect to a preferred port. This project uses :8888 so use that to avoid any issues. From there, press the Start Recording button to begin the sniffing process. From there, proceed with executing your .js script (Make sure the laravel server is running and that you are in the project's root). You should now see this in your Charles window:

That link is exactly what we need. You will notice that on my screen, SSL proxying is enabled. It needs to be for this to work. To enable it, simply right click the highlighted link in the left panel and press Enable SSL Proxying. (For me it's greyed out because my proxy service is actively running).

You should then see this button having been pressed.

If not, press it. It will enable SSL proxying directly, and then you can proceed. It is to be noted that once you have included your proxy connection in your Puppeteer code, it will not run unless the proxy service is on.. So if you're wondering why you're getting an error, it's (hopefully) that.

Unit Testing

I attempted to add Unit Testing with Jest but Puppeteer is already a good tool for automated testing so at the moment I did not implement it. Unit Testing for this project would require the refactoring of the original code in different functions, or "units", which can be tested individually,by mocking elements like the input, client-server communication, body parameters, etc.

A good unit-testing alternative, if it must be used, is PHPUnit, which should be included with this install of Puppeteer in the vendor/bin/phpunit directory. Likewise, the execute command for these types of tests is also vendor/bin/phpunit. You can further look into unit testing by checking out a small project i've made on this topic, which also uses PHP: https://github.com/x10combo/PHP-Calculator. The flights-prices-api project is quite more complex, but it should follow the same set of actions to a degree.