navbryce/warship-scraper

Scrapes warship data from Wikipedia and calcuates stats about the ships

Warship Scraper

Scrapes warship information from Wikipedia. Because all the information is user-sourced, most formatting is not consistent Wikipedia-page to page, making the task awkward. The scraper also calculates various stats about the ships and a similarity score between all the ships.

Background

I coded this app in 2017, while I was in high school. As a result, the code is not very idiomatic. Around 2020, it stopped working because the database was deleted and the stack was growing unsupported. In 2022, I switched to poetry for dependency management, packaged the app in Docker, and allocated a new database on Mongo Atlas, so I could re-deploy it on Heroku. Any of the changes I made in 2022 were deployment-related (and I didn't change the core of the projectt).

Stack

• Python (Scraper)
• Poetry
• Node.js and Express (HTTP API)
• MongoDB

Demo

See the frontend's repo to access the demo

warship-scraper's Issues

Fill-in Information

Some ships have information that is missing. The scraper should attempt to find this information through its class or a sister-ship.

Ships That Don't Work

Nagato

[Important] Error Checking

After scraping a ship, check for errors and verify the ship was scraped successfully before saving it to the database.

Date Range Problem

Dates in the format of {{year}}-{{other year}} will throw an error. Usually associated with "in-commission" property of ship.

Image Problems

• Weird caption on thumbnail image

• Wikipedia images have two sizes. Currently scrapes the smaller size. Should scrape both.

Nationality

Scrape ship's nationality and draw edge if two ships share a nationality.

Units with no Spaces

The scraper can't handle values in the following format: "{{value}}{{unit}}". Note: no space between value and unit.

Unit Conversions

Add unit converting for standard values AKA choose standard units. Already has been implemented for armor and gun size.

Better Error Logging System

Log scraping errors to file to make debugging easier

Fix Coding Conventions

As the title states, the coding conventions (the style of the code) are a little rough...because this was my first python project. Need to be fixed at some point.

Parallelize

Things that can/should be Parallelize:

• Script in ABoatScraping.ship_compare.ship_network_algo.run_algo_on_network.py