awarnes/mfc-store-scraper

Software for scraping wholesaler stores for MFC

mfc-store-scraper's Issues

Add Issue Template

Issues should generally look the same and include the necessary information to start triaging/work. Create an issue template that gets this information from filers.

Create generic project structure

In order to make sure we're able to wrap/import from other stores in the future, we need to generalize the code to allow us to easily create new store handlers without having to rewrite lots of code.

• Take a first stab at generalizing and breaking down the main.py file into constituent parts that make sense.

Add pylint to ensure all code follows guidelines

Add pylint to ensure that code follows proper guidelines for Python development.

• Add pylint (or similar linter)
• Lint all code and fix all issues
• Integrate with GitHub actions to lint project prior to PR merge and block when there are issues.

Make a self contained product class

This product class should know what is required for writing to a product endpoint (CSV or GraphQL) and could validate itself so that when data is added into it, we know when it's ready to write (minimum requirements have been met).

• Add generalized product class
• Has ability to validate itself before writing
• Can be used to generate both base products and variants
• Update hummingbird.format_data.format_products so that it uses the new class

Add ability to pull in real prices

Currently, the system just checks for what the public price is and not what the prices are for when we actually buy a product. We need to reformat the requests and such to be able to pull in that data (if possible). This will also allow us to accurately define the cost per item for internal analytics (what it actually costs us to sell).

• Log into hummingbird to get proper prices
• Use the variant_compare_at_price for the `price
• Use the new price returned as the cost_per_item

Update to make better command line tool

Right now you have to update code in main.py and run it to make changes. We should update the program to take in important command line information and make it generally more user friendly to run and work with.

• Add arg parsing to tool
• Allow for multiple sites to pull data from
• Allow for different arguments that can be useful for users
• Make sure it can run on Mac/Linux and Windows machines

Write integration tests

As we expand the scope of this repo we will want to include some level of integration tests that allow us to be sure everything is working well together.

Add Type hints to code

Once the code has been sufficiently generalized, go through and add the type hints to the code base so that everything is a little more clear.

• Type hints added to code base

Write full unit and integration tests for current system

With the current CSV generation system we'll want to make sure that the data we're pulling and generating is well tested so that we have to do as little manual work in the Shopify GUI as possible.

• Choose a test framework for Python
• Write necessary unit tests
• Write useful integration tests for system
• Integrate with GitHub actions to run tests for PRs and block if there are issues.

Expand and Simplify Developer On boarding

Everyone should be able to contribute simply to the project, and get the processes up and running quickly.

• Expand the documentation around building and running the project
• Add the proper requirements.txt file so that new developers can be sure to be on the right versions
• Add any install, utility, or management functions that would help developers build, test, and run the project.

Add Pull Request Template

Add a pull request template with the kinds of information we want to make sure is included in the pull request.