The argostime from m-rtijn

Add API so that 3rd parties can request Argostimè data

The idea is to add a simple JSON API via which Argostimè data can be requested, so that there is not only human-friendly access (via the main website) but also machine-friendly access (via the API)

Possible things we have to think about is:

rate limiting

Fix bug where normal price is shown as discount price in step graph

Incorrect prices shown for some Albert Heijn products.

At the time of writing this issue, the product "rode paprika" shows a price of 9.99, while the Albert Heijn site shows 1.09: https://argostime.mrtijn.nl/product/wi4117.

The issue also arises with AH Ribbelchips Naturel: https://argostime.mrtijn.nl/product/wi448475

Add links to the readme

Add link to the running instance
Add link(s) to the wiki (articles)
Add link to IRC chat (Howto: https://libera.chat/guides/webchat#linking-or-embedding)
Add link to the license file

Add documentation

Currently there is no proper documentation, good documentation could make the project more attractive to potential developers.

Documentation to add:

Setup guide (create venv, install pip packages in env, start flask etc)
Crawler guide (how to add a crawler to Argostime, this is a great entry point for new devs in my opinion)

AH incorrect price for some discounted products

E.g.

Will report a current price of €-1,00 and a lowest price of €9223372036854775808.00.

Fix Content-Security-Policy

The old CSP broke #24. So we need a new one

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

Gather historic data from archival websites

The idea is to get historic data from the (sporadic) archives that exist of product pages, on sites like web.archive.org or archive.is

Browser extension to add/view product

It would be nice to have an easier way to add products and view existing data when you are visiting the shop website.

I think we can achieve this with a separate browser extension for Chrome and Firefox. Feel free to share your thoughts on this idea here.

Some resources:

Use EAN codes to automatically link product offered by multiple shops

An EAN code uniquely identifies the product. Therefore if we find an EAN code at different shops, we know they are selling the same product!

Some products are already linked by hand but we should automate this!

Keep in mind though that EAN codes are not always available.

Add product volume to description

This affects the crawlers for:

Read enabled stores from configuration file

Jumbo missing on https://argostime.mrtijn.nl/

It looks like Jumbo is missing on https://argostime.mrtijn.nl/ but is available in this repository. Is https://argostime.mrtijn.nl/ in need of an update?

Add "Download the database" button

Add caching for graphs

Graphs only need to be updated once per day (except when you're debugging), so they are a good candidate for caching.

Make normal_price and discount_price optional in the CrawlResult class

See the discussion in #47

Add documentation in each crawler about which features it supports

The core of a crawler is to find a product name, product code and at least a normal price. Some things are optional (discount, EANs). There should be a clear format to comment in the source code of a specific crawler to show what it supports.

Improve new graphs on mobile devices

It's not taking up all space it can/should on my phone

Add support for Holland & Barret

Add support for kiwi-electronics.nl

Add support to also track availability

As an optional feature, a crawler may add availability information to a newly crawled price.

Availability information will be an extra property of the Price data model class. Possible values for the availability property are:

available directly from own stock ("uit voorraad leverbaar")
available from suppliers
not available, new stock expected
not available (no information about future restocking)
will never be available
no availability information (null)

Add support for Praxis

Add a logo

Have different color in the graphs if price was a discount price

Migrate to SQLAlchemy 2.0

https://docs.sqlalchemy.org/en/20/changelog/migration_20.html

Frontpage overhaul & add daily generated statistics

The current frontpage is getting really big because the product database has grown a lot.

Suggested changes:

Move the "currently discounted" to a separate page"
Generate daily statistics and show these on the front page

Daily statistics

Product count
Prices count
Offers count
Average/median offer age ("How long have we been tracking a product?")
Top most often discounted product (highest % of prices in db are labeled as "discount")
Top least often discounted products (might be boring)
Most volatile products (which product has the most changes in its price)
there are always more interesting statistics to show!

Fix some page load times

It takes a very long time to load the Albert Heijn page: https://argostime.mrtijn.nl/shop/1

I think we might be able to fix this using a more clever SQL query or adding indexes to the tables.

Resource on indexes: https://dev.mysql.com/doc/refman/8.0/en/mysql-indexes.html

Fix Karwei encoding issues with diacritics

The Karwei crawler doesn't like diacritics. Or maybe it's a more fundamental problem?

Change type annotations to Python 3.10

We're using python 3.10 instead of 3.8 now

Add support for Steam

Add sorting on tables

As the different shops can grow, and to keep oversight or search for deals, it can be useful to have some basic sorting on the tables in the shop overviews.

This can be achieved by using some Javascript, this one for example seems helpful:
https://github.com/LeeWannacott/table-sort-js

A visualization:

Implement discount detection for Simon Levelt

Hema crawler detects wrong prices + encoding issues

When a product has multiple options the crawler detects the wrong price, example
1+1 gratis is not detected, example.
Some characters are not parsed in the correct encoding format, example.

Parallelize update script

If we start a separate thread per webshop, we can still keep the amount of requests/hour low per webshop, but still in a faster way check all products in the database

Etos crawler can't detect discounts

I discovered that the Etos crawler cannot detect some discounts, they possibly added new types of discounts.

Types of discounts that do not work that I have found are:

20% korting
2e artikel 1,-

Bug for next weeks bonus in AH scraper

There is a bug in the AH scraper.

When a product is marked as being on discount next week, it is not parsed correctly.

Examples:

m-rtijn / argostime Goto Github PK

argostime's People

Contributors

Stargazers

Watchers

Forkers

argostime's Issues

Recommend Projects

Recommend Topics

Recommend Org