isaac-chung/nextpick

A web app that help users pick their next insta-vacation destinations.

NextPick

Pick your next insta-vacation

Website: nextpick.live

Presentation: Slides

What is NextPick?

There is an increasing portion of travellers saying that they use social media for planning their next trip, giving a rise to the trend of social media-inspired tourism in recent years. These social media-inspired travellers tend to go on "insta-holidays" (picturesque places), and "mini-vacations" (short duration). Closer-by locations are also favoured for travel time considerations. Combining the above creates the need for media and proximity driven searches for travel locations.

NextPick is my vision for addressing this need. It is a tool that recommends vacation spots based on an input image. Similar images, along with their locations, are returned based on cosine similarity between the images and geo-proximity to the user.

NextPick uses pre-trained deep learning models on the places365 dataset to generate image feature embeddings. These embeddings are compared to geo-tagged images from Flickr. Based on the image similarities and user-defined preferences, the top 5 locations are provided.

How to use NextPick

The landing page allows the user to choose an image, enter their current location, and select a proximity preference.

After an image is selected and the 'Go' button is clicked, the selected image and the entered location are shown.

This is followed by 5 images, along with their cosine difference to the input image and the distance to the current location. Clicking the address would lead to a Google search of the address.

And finally, the locations of those images are visualized on a map. Labels for each data point is shown on hover. The red dot is the user's entered location.

The purple dots are recommended locations.

The user can zoom in and out of the Plotly map.

Local setup

To run a local copy of NextPick, clone the master branch of the repository with

git clone https://github.com/isaac-chung/insight.git

Then, create a virtual environment and have all the dependencies in requirements.txt installed. With Anaconda3, we can use the following to create one called insight with Python 3.8 and all packages installed.

conda create -n insight python=3.8
source activate insight
pip install -r requirements.txt

I am currently keeping the image data on this repository. Once it's downloaded, place the folder at the root level of the insight project folder, and rename it as data. If this link is broken, you can follow this Jupyter notebook to retrieve your own geotagged imaged data using the Flickr API.

Also, in config.py, change the variable APP_PATH to the path of the app folder.

The app can be run locally using

python server.py

Navigate via your web browser to http://127.0.0.1:5000/. This version of the app is tested on Windows 10 and Ubuntu 18.04 (WSL).

Unit Tests

NextPick comes with a few unit tests, which can be extended in a later time. To run these tests, first modify line 16 image_search.py from import NextPick.config as cfg to import config as cfg. Then navigate to the first NextPick directory (nextpick/NextPick-app/NextPick), and type the following in the command line:

python -m unittest discover -p '*_test.py'

Directory Files

|-- NextPick-app
    (Holds the standalone web app)
    |-- NextPick
        |-- (where 'data' should be)
        |-- NextPick
            (folder for image processing, search, visualization functions)
            |-- image_search.py
            |-- ImageDataset.py
            |-- plotly_map.py
        |-- static
            (asset folder for the web app)
        |-- templates
            (html templates folder for the web app)
        |-- test
            (unit tests folder)
        |-- config.py
        |-- requirements.txt
        |-- server.py
        |-- setup.py
    |-- run.py
|-- notebooks
    (notebooks folder for data acquisition and exploratory data analyses)
|-- places365
    (asset folder from the places365 repo)
|-- scripts
    (misc scripts)
|-- static_img
    (image folder for README.md)

What's next?

While it is true that there are small adjustments that can be made to the web app, a logical next step for this project is to dockerize the app. Docker enables developers to easily pack, ship, and run any application as a lightweight, portable, self-sufficient container, which can run virtually anywhere (ZDNet). Docker containers are also easy to deploy in a cloud.

Installing Docker

To get Docker running at the time of writing this on my Windows 10 Home machine, version 1909 (OS Build 18363.1082), I followed these steps. I also read this issue when setting the WSL version to 2, and this issue when starting up Docker Quickstart.

nextpick's Issues

Addresses don't match the image

Settings:
same image (banff, AB, middle image); near; varying locations (Toronto, Amsterdam, Hong Kong).

Same images would be recommended for different locations, but the addresses would change. There is a disconnect here.

High latency when reverse geocoding

GeoPy takes about 0.5 seconds per request. Can see if there are any free batch/bulk reverse coding APIs.

Use a more refined annoy search tree

The new tree annoy_idx_2.annoy is uploaded. Just need to update the scripts to point to it and use 132 as the build param when loading the annoy index.

Unit tests being run twice

Following the instructions in README, the unit tests will be run twice, i.e. there are 5 tests, and 10 will be run. Total run time will be doubled. Want to run them only once.

How long is the wait?

Add a throbber / spinning circle waiting thing / progress bar type of visual while processing the input image.

Clarify search metric used in Annoy

It's more so 'cosine distance' (euclidean distance of the normalized (unit) vectors) over 'cosine difference', as described in the Annoy repo and confirmed in the user group discussion. Would need changes in 'README' and 'results.html'.

unit tests

Currently, before running unit tests, the test files needed to be moved to the parent folder (from NextPick/test to NextPick). Want to be able to run the these test without doing so.

Links of recommended pictures

Add hyperlink to the recommended pictures. Clicking would have the same effect as if that picture was input to the web app.