This is website crawler which uses Geb to read data from Auto motor und sport website, and save it in local Mongodb. It uses only Chrome to perform this crawling. This project used Example Geb and Gradle project as it base.
Crawler gathers data from Supertests. This tests are conducted on Nurburgring Nordschleife ("North Loop") and on Hockenheimring short course tracks. Gathered data contains:
- Car data - make, model, production years, gearbox type, layout (AWD, FWD, RWD), weight, engine power, engine torque
- Test info - driver, test date, url
- Test results - Nordschleife laptime, Hockenheim laptime, acceleration times 0-100 km/h and 0-200 km/h
Supertest contains more data, but currently only data mentioned above is crawled. Information about tyres and some test dates was added manually. If there was no information about tyres in article I checked other tests from Auto motor und sport or looked for articles in internet.
Crawler has these scripts:
- importLinks - it goes through paged list of supertests and saves url's
- verifyLinks - it goes through saved url's and check's which supertests have tests results (some supertests are testing SUV/terrain cars, and don't contain track data; some tests of sports cars don't have relevant data in test results but in text, see below in Problems)
- readTestData - it goes through verified links, checks if data exists once more and reads it
- readMissingTestData - it goes through tests which were marked as not having relevant data, missing data will be added manually, this tests are mentioned below, this tests were picked manually, see below in Problems
- addMissingData - added missing data from json (tyres, tyres spec, source of information about tyres, test date according to test title)
- generateWikiTable - generate table for wiki
You run them in this order:
gradle importLinks
gradle verifyLinks
gradle readTestData
gradle readMissingTestData
gradle addMissingData
gradle generateWikiTable
Script importLinks inserts links to new tests to collection links. verifyLinks only updates data. readTestData inserts new data, and can be restarted if it fails. It starts where it stopped before. If you want to read data from scratch you must drop collection named results and links.
Data from Supertests especially Nordschleife laptimes are quite reliable. Most of tests are performed by one driver, Horst von Saurma and cars are in factory specification including tyres and equipment. Automakers have their 'own definition' of factory/street-legal specification. This tests are published in sport auto magazine
I wanted to become more familiar with Geb and use Mongodb which I hadn't chance to use before.
This tests results were added manually:
- McLaren 650s Spider - 9266605 - Nurburgring data on image
- BMW M5 Competition - 8911761 - Nurburgring data on image and in text
- Audi TT RS - 1747833 - Nurburgring data on image
- Ferrari 599 GTB - 1041324 - Nurburgring data on image and in text
- Porsche 911 GT2 - 1041363 - Nurburgring data on image and in text
- Nissan GT-R - 1347840 - Nurburgring data on image and in text
- Lamborghini Gallardo LP560-4 - 1041516 - Nurburgring data on image and in text
- Ferrari F430 Scuderia - 1041447 - Nurburgring data on image and in text
- Lamborghini Murcielago LP640 - 1041040 - Nurburgring data on image and in text
- Porsche 911 GT3 - 1040952 - Nurburgring data on image and in text
- Pagani Zonda S - 1036024 - Nurburgring data in text
This tests was added manually (whole test):
- Audi RS4 - 1036092 - Nurburgring data in test section
This tests don't have any test results:
- Renault Megane Sport Trophy - 1041029 - no data at all
- Aston Martin V12 Vanquish - 1036049 - no data at all
- Ferrari 550 Maranello - 1041307 - no data at all
- Porsche 918 has got power only from combustion engine
- gradle task have some steps in common, it can be refactored
- no tyre types (semi, UHP, etc)
- http://m.forocoches.com/foro/showthread.php?t=195127
- https://www.gtplanet.net/forum/threads/official-sportauto-hockenheim-laptimes.82508/
- https://www.scoobynet.com/other-marques-33/537667-official-sportauto-hockenheim-laptimes.html
- https://www.scoobynet.com/scoobynet-general-1/843312-best-nurburgring-lap-times.html
- https://www.scoobynet.com/scoobynet-general-1/395646-lap-times-round-nurburgring-sp.html