corb1999/ski_shop

personal-use web crawling project/practice

ski_shop's Issues

prototype eda with database connection

test out the sqlite database and create a script that can query from the database, do some additional processing, and then visualize and analyze the results.

build initial crawler script

finish building the script that does the following:

• makes a list of urls to crawl, each url another page. use purr to generate the list
• do an initial page read, then calculate how many total pages need to be read
• write a function to read the html, then parse it into a tidy dataframe and include a Sys.sleep() to not spam
• minor cleaning on the pages then compiling them all into one dataframe
• append a time stamp to the result then export to a dropzone

enhancements to sql view

add sql code to the view which does...

• make a series of indicator fields to detect is things like skis, bindings, helmets, jackets, gloves, masks, bindings, boots, are in the product name
• remove no longer necessary raw columns like regular price and discount price that are just pure text values
• price differences between regular and discount, nominal differences and percent difference

problem with discounted percent calculation or data

when doing eda and scatterplot of all the discount percent values, found a number of records that show as having zero discount percent even though the nominal discount calc is a negative value and the regular and discount values are not equal. doesnt seem to be a rounding error. may need to be fixed on the database end. run this snippet to inspect when this is occurring: df %>% filter(delta_price_percent == 0) %>% View()

run the crawler multiple times in a month, then build unionizer etl

after completing issue #1 then build a script that can consume any and all crawler runs into a single unified dataframe that can be then later be analyzed